Oct 30 13:00:20.507827 ip-10-0-134-31 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 13:00:20.507838 ip-10-0-134-31 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 13:00:20.507848 ip-10-0-134-31 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 13:00:20.508169 ip-10-0-134-31 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 13:00:30.535472 ip-10-0-134-31 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 13:00:30.535511 ip-10-0-134-31 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot cff3fcb70044495ebdafbbea1d4e2f7f --
Oct 30 13:02:18.001559 ip-10-0-134-31 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 13:02:18.434169 ip-10-0-134-31 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 13:02:18.434169 ip-10-0-134-31 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 13:02:18.434169 ip-10-0-134-31 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 13:02:18.434169 ip-10-0-134-31 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 13:02:18.440844 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440752    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 13:02:18.440844 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440845    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 13:02:18.440844 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440849    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 13:02:18.440942 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440853    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 13:02:18.440942 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440856    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 13:02:18.440942 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440858    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 13:02:18.440942 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440861    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 13:02:18.440942 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440865    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 13:02:18.440942 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440868    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 13:02:18.440942 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440871    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 13:02:18.440942 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440874    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 13:02:18.440942 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440876    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 13:02:18.440942 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440879    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 13:02:18.440942 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440881    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 13:02:18.440942 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440884    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 13:02:18.440942 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440887    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 13:02:18.440942 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440889    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 13:02:18.440942 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440892    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 13:02:18.440942 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440894    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 13:02:18.440942 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440897    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 13:02:18.440942 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440900    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 13:02:18.440942 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440902    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 13:02:18.441402 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440907    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 13:02:18.441402 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440912    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 13:02:18.441402 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440918    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 13:02:18.441402 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440921    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 13:02:18.441402 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440925    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 13:02:18.441402 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440928    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 13:02:18.441402 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440930    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 13:02:18.441402 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440933    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 13:02:18.441402 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440935    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 13:02:18.441402 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440938    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 13:02:18.441402 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440940    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 13:02:18.441402 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440943    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 13:02:18.441402 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440946    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 13:02:18.441402 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440949    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 13:02:18.441402 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440952    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 13:02:18.441402 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440955    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 13:02:18.441402 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440958    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 13:02:18.441402 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440960    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 13:02:18.441402 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440963    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 13:02:18.441402 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440965    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 13:02:18.441864 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440968    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 13:02:18.441864 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440970    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 13:02:18.441864 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440973    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 13:02:18.441864 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440975    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 13:02:18.441864 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440978    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 13:02:18.441864 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440988    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 13:02:18.441864 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440991    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 13:02:18.441864 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440993    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 13:02:18.441864 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440996    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 13:02:18.441864 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.440998    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 13:02:18.441864 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441001    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 13:02:18.441864 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441004    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 13:02:18.441864 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441007    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 13:02:18.441864 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441010    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 13:02:18.441864 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441013    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 13:02:18.441864 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441016    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 13:02:18.441864 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441018    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 13:02:18.441864 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441021    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 13:02:18.441864 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441024    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 13:02:18.442329 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441027    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 13:02:18.442329 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441029    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 13:02:18.442329 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441032    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 13:02:18.442329 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441034    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 13:02:18.442329 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441036    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 13:02:18.442329 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441039    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 13:02:18.442329 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441041    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 13:02:18.442329 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441044    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 13:02:18.442329 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441052    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 13:02:18.442329 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441055    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 13:02:18.442329 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441058    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 13:02:18.442329 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441060    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 13:02:18.442329 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441063    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 13:02:18.442329 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441065    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 13:02:18.442329 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441067    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 13:02:18.442329 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441070    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 13:02:18.442329 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441072    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 13:02:18.442329 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441075    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 13:02:18.442329 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441077    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 13:02:18.442817 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441086    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 13:02:18.442817 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441089    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 13:02:18.442817 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441092    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 13:02:18.442817 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441094    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 13:02:18.442817 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441096    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 13:02:18.442817 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441100    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 13:02:18.442817 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441103    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 13:02:18.442817 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441106    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 13:02:18.442817 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441109    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 13:02:18.442817 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441112    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 13:02:18.442817 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441114    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 13:02:18.442817 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441117    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 13:02:18.442817 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441121    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 13:02:18.442817 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441137    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 13:02:18.442817 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441141    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 13:02:18.442817 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441144    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 13:02:18.442817 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441146    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 13:02:18.442817 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441149    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 13:02:18.442817 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441153    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 13:02:18.442817 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441155    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 13:02:18.443299 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441158    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 13:02:18.443299 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441166    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 13:02:18.443299 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441169    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 13:02:18.443299 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441172    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 13:02:18.443299 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441174    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 13:02:18.443299 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.441178    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 13:02:18.443299 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442148    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 13:02:18.443299 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442156    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 13:02:18.443299 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442160    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 13:02:18.443299 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442162    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 13:02:18.443299 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442165    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 13:02:18.443299 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442168    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 13:02:18.443299 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442170    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 13:02:18.443299 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442179    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 13:02:18.443299 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442183    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 13:02:18.443299 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442186    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 13:02:18.443299 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442188    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 13:02:18.443299 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442191    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 13:02:18.443299 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442194    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 13:02:18.443773 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442197    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 13:02:18.443773 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442200    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 13:02:18.443773 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442203    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 13:02:18.443773 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442206    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 13:02:18.443773 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442209    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 13:02:18.443773 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442212    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 13:02:18.443773 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442215    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 13:02:18.443773 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442217    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 13:02:18.443773 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442234    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 13:02:18.443773 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442239    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 13:02:18.443773 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442242    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 13:02:18.443773 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442245    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 13:02:18.443773 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442249    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 13:02:18.443773 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442251    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 13:02:18.443773 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442254    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 13:02:18.443773 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442257    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 13:02:18.443773 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442260    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 13:02:18.443773 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442263    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 13:02:18.443773 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442266    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 13:02:18.443773 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442268    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 13:02:18.444275 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442271    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 13:02:18.444275 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442274    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 13:02:18.444275 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442277    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 13:02:18.444275 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442279    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 13:02:18.444275 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442282    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 13:02:18.444275 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442285    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 13:02:18.444275 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442287    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 13:02:18.444275 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442290    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 13:02:18.444275 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442293    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 13:02:18.444275 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442297    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 13:02:18.444275 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442302    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 13:02:18.444275 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442305    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 13:02:18.444275 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442310    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 13:02:18.444275 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442314    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 13:02:18.444275 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442318    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 13:02:18.444275 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442321    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 13:02:18.444275 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442323    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 13:02:18.444275 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442326    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 13:02:18.444275 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442329    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 13:02:18.444721 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442332    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 13:02:18.444721 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442335    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 13:02:18.444721 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442338    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 13:02:18.444721 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442341    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 13:02:18.444721 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442344    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 13:02:18.444721 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442346    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 13:02:18.444721 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442349    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 13:02:18.444721 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442352    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 13:02:18.444721 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442354    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 13:02:18.444721 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442357    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 13:02:18.444721 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442360    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 13:02:18.444721 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442362    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 13:02:18.444721 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442365    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 13:02:18.444721 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442368    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 13:02:18.444721 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442371    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 13:02:18.444721 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442374    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 13:02:18.444721 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442377    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 13:02:18.444721 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442380    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 13:02:18.444721 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442382    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 13:02:18.444721 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442385    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 13:02:18.445211 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442388    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 13:02:18.445211 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442391    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 13:02:18.445211 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442394    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 13:02:18.445211 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442396    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 13:02:18.445211 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442400    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 13:02:18.445211 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442402    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 13:02:18.445211 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442405    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 13:02:18.445211 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442408    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 13:02:18.445211 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442410    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 13:02:18.445211 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442413    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 13:02:18.445211 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442415    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 13:02:18.445211 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442418    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 13:02:18.445211 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442420    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 13:02:18.445211 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442423    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 13:02:18.445211 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442425    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 13:02:18.445211 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442428    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 13:02:18.445211 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442430    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 13:02:18.445211 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442433    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 13:02:18.445211 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442435    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 13:02:18.445211 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442438    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 13:02:18.445711 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442440    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 13:02:18.445711 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442444    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 13:02:18.445711 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442447    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 13:02:18.445711 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442449    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 13:02:18.445711 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442452    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 13:02:18.445711 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442455    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 13:02:18.445711 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442458    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 13:02:18.445711 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442461    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 13:02:18.445711 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442463    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 13:02:18.445711 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442466    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 13:02:18.445711 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442468    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 13:02:18.445711 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442471    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 13:02:18.445711 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442473    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 13:02:18.445711 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.442481    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 13:02:18.445711 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443034    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 13:02:18.445711 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443040    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 13:02:18.445711 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443043    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 13:02:18.445711 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443046    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 13:02:18.445711 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443049    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 13:02:18.446183 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443052    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 13:02:18.446183 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443054    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 13:02:18.446183 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443057    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 13:02:18.446183 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443060    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 13:02:18.446183 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443063    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 13:02:18.446183 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443066    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 13:02:18.446183 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443069    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 13:02:18.446183 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443071    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 13:02:18.446183 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443074    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 13:02:18.446183 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443077    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 13:02:18.446183 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443079    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 13:02:18.446183 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443082    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 13:02:18.446183 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443084    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 13:02:18.446183 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443087    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 13:02:18.446183 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443090    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 13:02:18.446183 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443092    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 13:02:18.446183 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443095    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 13:02:18.446183 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443098    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 13:02:18.446183 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443103    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 13:02:18.446639 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443107    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 13:02:18.446639 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443111    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 13:02:18.446639 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443114    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 13:02:18.446639 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443117    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 13:02:18.446639 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443120    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 13:02:18.446639 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443122    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 13:02:18.446639 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443125    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 13:02:18.446639 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443140    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 13:02:18.446639 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443144    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 13:02:18.446639 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443148    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 13:02:18.446639 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443150    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 13:02:18.446639 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443153    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 13:02:18.446639 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443155    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 13:02:18.446639 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443158    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 13:02:18.446639 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443161    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 13:02:18.446639 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443163    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 13:02:18.446639 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443166    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 13:02:18.446639 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443168    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 13:02:18.446639 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443171    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 13:02:18.446639 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443174    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 13:02:18.447225 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443177    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 13:02:18.447225 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443179    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 13:02:18.447225 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443182    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 13:02:18.447225 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443185    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 13:02:18.447225 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443188    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 13:02:18.447225 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443191    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 13:02:18.447225 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443193    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 13:02:18.447225 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443196    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 13:02:18.447225 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443199    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 13:02:18.447225 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443201    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 13:02:18.447225 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443204    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 13:02:18.447225 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443207    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 13:02:18.447225 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443209    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 13:02:18.447225 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443212    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 13:02:18.447225 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443215    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 13:02:18.447225 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443217    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 13:02:18.447225 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443220    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 13:02:18.447225 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443223    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 13:02:18.447225 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443225    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 13:02:18.447225 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443228    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 13:02:18.447694 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443230    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 13:02:18.447694 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443233    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 13:02:18.447694 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443236    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 13:02:18.447694 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443239    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 13:02:18.447694 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443241    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 13:02:18.447694 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443244    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 13:02:18.447694 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443247    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 13:02:18.447694 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443249    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 13:02:18.447694 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443252    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 13:02:18.447694 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443255    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 13:02:18.447694 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443257    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 13:02:18.447694 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443260    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 13:02:18.447694 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443262    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 13:02:18.447694 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443264    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 13:02:18.447694 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443268    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 13:02:18.447694 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443270    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 13:02:18.447694 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443273    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 13:02:18.447694 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443275    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 13:02:18.447694 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443278    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 13:02:18.447694 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443280    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 13:02:18.448177 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443283    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 13:02:18.448177 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443285    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 13:02:18.448177 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443288    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 13:02:18.448177 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443290    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 13:02:18.448177 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443292    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 13:02:18.448177 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443295    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 13:02:18.448177 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443297    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 13:02:18.448177 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443300    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 13:02:18.448177 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443303    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 13:02:18.448177 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443305    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 13:02:18.448177 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443308    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 13:02:18.448177 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443311    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 13:02:18.448177 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443313    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 13:02:18.448177 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443316    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 13:02:18.448177 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443320    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 13:02:18.448177 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443324    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 13:02:18.448177 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443326    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 13:02:18.448177 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443329    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 13:02:18.448177 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443331    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 13:02:18.448627 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443333    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 13:02:18.448627 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443336    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 13:02:18.448627 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.443338    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 13:02:18.448627 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444003    2576 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 13:02:18.448627 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444012    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 13:02:18.448627 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444019    2576 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 13:02:18.448627 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444025    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 13:02:18.448627 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444030    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 13:02:18.448627 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444033    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 13:02:18.448627 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444037    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 13:02:18.448627 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444042    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 13:02:18.448627 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444045    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 13:02:18.448627 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444049    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 13:02:18.448627 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444052    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 13:02:18.448627 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444056    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 13:02:18.448627 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444059    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 13:02:18.448627 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444062    2576 flags.go:64] FLAG: --cgroup-root=""
Oct 30 13:02:18.448627 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444065    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 13:02:18.448627 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444068    2576 flags.go:64] FLAG: --client-ca-file=""
Oct 30 13:02:18.448627 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444071    2576 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 13:02:18.448627 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444074    2576 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 13:02:18.448627 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444079    2576 flags.go:64] FLAG: --cluster-domain=""
Oct 30 13:02:18.448627 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444082    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 13:02:18.449258 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444085    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 13:02:18.449258 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444089    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 13:02:18.449258 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444093    2576 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 13:02:18.449258 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444100    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 13:02:18.449258 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444103    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 13:02:18.449258 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444107    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 13:02:18.449258 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444110    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 13:02:18.449258 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444113    2576 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 13:02:18.449258 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444117    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 13:02:18.449258 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444120    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 13:02:18.449258 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444123    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 13:02:18.449258 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444144    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 13:02:18.449258 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444149    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 13:02:18.449258 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444152    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 13:02:18.449258 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444155    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 13:02:18.449258 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444159    2576 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 13:02:18.449258 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444162    2576 flags.go:64] FLAG: --enable-server="true"
Oct 30 13:02:18.449258 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444165    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 13:02:18.449258 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444170    2576 flags.go:64] FLAG: --event-burst="100"
Oct 30 13:02:18.449258 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444173    2576 flags.go:64] FLAG: --event-qps="50"
Oct 30 13:02:18.449258 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444176    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 13:02:18.449258 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444179    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 13:02:18.449258 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444182    2576 flags.go:64] FLAG: --eviction-hard=""
Oct 30 13:02:18.449258 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444186    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 13:02:18.449841 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444190    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 13:02:18.449841 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444193    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 13:02:18.449841 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444197    2576 flags.go:64] FLAG: --eviction-soft=""
Oct 30 13:02:18.449841 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444200    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 13:02:18.449841 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444202    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 13:02:18.449841 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444205    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 13:02:18.449841 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444208    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 13:02:18.449841 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444211    2576 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 13:02:18.449841 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444214    2576 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 13:02:18.449841 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444217    2576 flags.go:64] FLAG: --feature-gates=""
Oct 30 13:02:18.449841 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444226    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 13:02:18.449841 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444231    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 13:02:18.449841 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444234    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 13:02:18.449841 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444238    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 13:02:18.449841 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444242    2576 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 13:02:18.449841 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444245    2576 flags.go:64] FLAG: --help="false"
Oct 30 13:02:18.449841 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444248    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-134-31.ec2.internal"
Oct 30 13:02:18.449841 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444251    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 13:02:18.449841 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444255    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 13:02:18.449841 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444258    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 13:02:18.449841 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444261    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 13:02:18.449841 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444265    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 13:02:18.449841 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444268    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 13:02:18.449841 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444271    2576 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 13:02:18.450423 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444274    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 13:02:18.450423 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444277    2576 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 13:02:18.450423 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444280    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 13:02:18.450423 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444284    2576 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 13:02:18.450423 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444287    2576 flags.go:64] FLAG: --kube-reserved=""
Oct 30 13:02:18.450423 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444290    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 13:02:18.450423 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444293    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 13:02:18.450423 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444296    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 13:02:18.450423 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444299    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 13:02:18.450423 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444302    2576 flags.go:64] FLAG: --lock-file=""
Oct 30 13:02:18.450423 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444305    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 13:02:18.450423 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444308    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 13:02:18.450423 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444312    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 13:02:18.450423 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444318    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 13:02:18.450423 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444321    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 13:02:18.450423 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444324    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 13:02:18.450423 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444327    2576 flags.go:64] FLAG: --logging-format="text"
Oct 30 13:02:18.450423 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444330    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 13:02:18.450423 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444333    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 13:02:18.450423 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444338    2576 flags.go:64] FLAG: --manifest-url=""
Oct 30 13:02:18.450423 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444341    2576 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 13:02:18.450423 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444346    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 13:02:18.450423 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444349    2576 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 13:02:18.450423 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444354    2576 flags.go:64] FLAG: --max-pods="110"
Oct 30 13:02:18.450423 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444357    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 13:02:18.451002 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444361    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 13:02:18.451002 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444364    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 13:02:18.451002 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444367    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 13:02:18.451002 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444370    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 13:02:18.451002 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444373    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 13:02:18.451002 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444378    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 13:02:18.451002 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444386    2576 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 13:02:18.451002 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444389    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 13:02:18.451002 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444392    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 13:02:18.451002 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444395    2576 flags.go:64] FLAG: --pod-cidr=""
Oct 30 13:02:18.451002 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444399    2576 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 13:02:18.451002 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444402    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 13:02:18.451002 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444405    2576 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 13:02:18.451002 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444408    2576 flags.go:64] FLAG: --port="10250"
Oct 30 13:02:18.451002 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444411    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 13:02:18.451002 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444414    2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c1fca1c339c24a35"
Oct 30 13:02:18.451002 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444418    2576 flags.go:64] FLAG: --qos-reserved=""
Oct 30 13:02:18.451002 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444421    2576 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 13:02:18.451002 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444424    2576 flags.go:64] FLAG: --register-node="true"
Oct 30 13:02:18.451002 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444427    2576 flags.go:64] FLAG: --register-with-taints=""
Oct 30 13:02:18.451002 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444431    2576 flags.go:64] FLAG: --registry-burst="10"
Oct 30 13:02:18.451002 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444434    2576 flags.go:64] FLAG: --registry-qps="5"
Oct 30 13:02:18.451002 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444437    2576 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 13:02:18.451002 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444440    2576 flags.go:64] FLAG: --reserved-memory=""
Oct 30 13:02:18.451002 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444445    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 13:02:18.451002 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444448    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 13:02:18.451660 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444452    2576 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 13:02:18.451660 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444455    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 13:02:18.451660 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444458    2576 flags.go:64] FLAG: --runonce="false"
Oct 30 13:02:18.451660 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444461    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 13:02:18.451660 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444464    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 13:02:18.451660 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444468    2576 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 13:02:18.451660 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444471    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 13:02:18.451660 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444474    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 13:02:18.451660 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444477    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 13:02:18.451660 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444480    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 13:02:18.451660 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444484    2576 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 13:02:18.451660 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444489    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 13:02:18.451660 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444492    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 13:02:18.451660 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444495    2576 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 13:02:18.451660 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444498    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 13:02:18.451660 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444501    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 13:02:18.451660 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444504    2576 flags.go:64] FLAG: --system-cgroups=""
Oct 30 13:02:18.451660 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444507    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 13:02:18.451660 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444513    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 13:02:18.451660 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444516    2576 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 13:02:18.451660 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444518    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 13:02:18.451660 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444524    2576 flags.go:64] FLAG: --tls-min-version=""
Oct 30 13:02:18.451660 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444527    2576 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 13:02:18.451660 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444529    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 13:02:18.451660 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444532    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 13:02:18.452261 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444535    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 13:02:18.452261 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444538    2576 flags.go:64] FLAG: --v="2"
Oct 30 13:02:18.452261 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444543    2576 flags.go:64] FLAG: --version="false"
Oct 30 13:02:18.452261 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444547    2576 flags.go:64] FLAG: --vmodule=""
Oct 30 13:02:18.452261 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444551    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 13:02:18.452261 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.444555    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 13:02:18.452261 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444691    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 13:02:18.452261 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444696    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 13:02:18.452261 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444699    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 13:02:18.452261 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444703    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 13:02:18.452261 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444705    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 13:02:18.452261 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444708    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 13:02:18.452261 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444712    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 13:02:18.452261 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444715    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 13:02:18.452261 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444729    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 13:02:18.452261 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444733    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 13:02:18.452261 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444737    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 13:02:18.452261 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444740    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 13:02:18.452261 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444745    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 13:02:18.452261 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444747    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 13:02:18.452737 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444750    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 13:02:18.452737 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444753    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 13:02:18.452737 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444756    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 13:02:18.452737 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444758    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 13:02:18.452737 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444761    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 13:02:18.452737 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444764    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 13:02:18.452737 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444766    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 13:02:18.452737 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444769    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 13:02:18.452737 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444771    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 13:02:18.452737 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444774    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 13:02:18.452737 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444777    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 13:02:18.452737 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444779    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 13:02:18.452737 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444782    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 13:02:18.452737 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444784    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 13:02:18.452737 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444787    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 13:02:18.452737 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444789    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 13:02:18.452737 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444792    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 13:02:18.452737 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444794    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 13:02:18.452737 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444797    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 13:02:18.452737 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444800    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 13:02:18.453295 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444803    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 13:02:18.453295 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444805    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 13:02:18.453295 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444808    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 13:02:18.453295 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444810    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 13:02:18.453295 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444813    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 13:02:18.453295 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444815    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 13:02:18.453295 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444818    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 13:02:18.453295 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444821    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 13:02:18.453295 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444824    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 13:02:18.453295 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444827    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 13:02:18.453295 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444831    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 13:02:18.453295 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444833    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 13:02:18.453295 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444835    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 13:02:18.453295 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444838    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 13:02:18.453295 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444841    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 13:02:18.453295 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444843    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 13:02:18.453295 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444846    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 13:02:18.453295 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444849    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 13:02:18.453295 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444853    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 13:02:18.453740 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444858    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 13:02:18.453740 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444861    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 13:02:18.453740 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444864    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 13:02:18.453740 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444868    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 13:02:18.453740 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444870    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 13:02:18.453740 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444874    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 13:02:18.453740 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444879    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 13:02:18.453740 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444881    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 13:02:18.453740 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444884    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 13:02:18.453740 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444887    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 13:02:18.453740 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444890    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 13:02:18.453740 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444892    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 13:02:18.453740 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444895    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 13:02:18.453740 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444898    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 13:02:18.453740 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444901    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 13:02:18.453740 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444903    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 13:02:18.453740 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444906    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 13:02:18.453740 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444908    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 13:02:18.453740 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444911    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 13:02:18.454196 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444913    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 13:02:18.454196 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444917    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 13:02:18.454196 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444919    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 13:02:18.454196 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444922    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 13:02:18.454196 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444926    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 13:02:18.454196 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444929    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 13:02:18.454196 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444932    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 13:02:18.454196 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444934    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 13:02:18.454196 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444937    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 13:02:18.454196 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444940    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 13:02:18.454196 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444942    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 13:02:18.454196 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444945    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 13:02:18.454196 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444947    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 13:02:18.454196 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444950    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 13:02:18.454196 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444953    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 13:02:18.454196 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444955    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 13:02:18.454196 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444958    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 13:02:18.454196 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444960    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 13:02:18.454196 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444963    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 13:02:18.454196 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444966    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 13:02:18.454693 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444968    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 13:02:18.454693 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444971    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 13:02:18.454693 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444973    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 13:02:18.454693 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444976    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 13:02:18.454693 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444979    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 13:02:18.454693 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444982    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 13:02:18.454693 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444985    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 13:02:18.454693 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444987    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 13:02:18.454693 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444990    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 13:02:18.454693 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444992    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 13:02:18.454693 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444995    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 13:02:18.454693 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.444997    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 13:02:18.454693 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445000    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 13:02:18.454693 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445002    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 13:02:18.454693 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.445029    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 13:02:18.454693 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445072    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 13:02:18.455074 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445077    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 13:02:18.455074 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445082    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 13:02:18.455074 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445085    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 13:02:18.455074 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445088    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 13:02:18.455074 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445090    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 13:02:18.455074 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445093    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 13:02:18.455074 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445095    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 13:02:18.455074 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445098    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 13:02:18.455074 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445101    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 13:02:18.455074 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445104    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 13:02:18.455074 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445106    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 13:02:18.455074 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445109    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 13:02:18.455074 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445111    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 13:02:18.455074 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445114    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 13:02:18.455074 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445117    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 13:02:18.455074 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445119    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 13:02:18.455074 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445122    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 13:02:18.455074 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445124    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 13:02:18.455074 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445138    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 13:02:18.455074 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445154    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 13:02:18.455558 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445157    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 13:02:18.455558 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445167    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 13:02:18.455558 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445170    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 13:02:18.455558 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445174    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 13:02:18.455558 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445176    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 13:02:18.455558 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445179    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 13:02:18.455558 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445181    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 13:02:18.455558 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445184    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 13:02:18.455558 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445186    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 13:02:18.455558 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445189    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 13:02:18.455558 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445191    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 13:02:18.455558 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445195    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 13:02:18.455558 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445198    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 13:02:18.455558 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445202    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 13:02:18.455558 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445205    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 13:02:18.455558 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445207    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 13:02:18.455558 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445210    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 13:02:18.455558 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445215    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 13:02:18.455558 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445218    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 13:02:18.456007 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445221    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 13:02:18.456007 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445224    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 13:02:18.456007 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445226    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 13:02:18.456007 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445229    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 13:02:18.456007 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445232    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 13:02:18.456007 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445234    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 13:02:18.456007 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445237    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 13:02:18.456007 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445239    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 13:02:18.456007 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445242    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 13:02:18.456007 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445245    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 13:02:18.456007 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445247    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 13:02:18.456007 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445250    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 13:02:18.456007 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445252    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 13:02:18.456007 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445255    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 13:02:18.456007 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445257    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 13:02:18.456007 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445259    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 13:02:18.456007 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445262    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 13:02:18.456007 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445264    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 13:02:18.456007 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445267    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 13:02:18.456472 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445269    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 13:02:18.456472 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445272    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 13:02:18.456472 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445274    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 13:02:18.456472 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445277    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 13:02:18.456472 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445279    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 13:02:18.456472 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445282    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 13:02:18.456472 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445285    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 13:02:18.456472 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445289    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 13:02:18.456472 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445292    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 13:02:18.456472 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445294    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 13:02:18.456472 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445297    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 13:02:18.456472 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445301    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 13:02:18.456472 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445304    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 13:02:18.456472 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445307    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 13:02:18.456472 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445310    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 13:02:18.456472 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445312    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 13:02:18.456472 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445314    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 13:02:18.456472 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445317    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 13:02:18.456472 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445320    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 13:02:18.456472 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445322    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 13:02:18.456932 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445325    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 13:02:18.456932 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445327    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 13:02:18.456932 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445330    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 13:02:18.456932 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445332    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 13:02:18.456932 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445335    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 13:02:18.456932 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445337    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 13:02:18.456932 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445340    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 13:02:18.456932 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445343    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 13:02:18.456932 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445348    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 13:02:18.456932 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445352    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 13:02:18.456932 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445356    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 13:02:18.456932 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445359    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 13:02:18.456932 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445362    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 13:02:18.456932 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445366    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 13:02:18.456932 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445369    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 13:02:18.456932 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445374    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 13:02:18.456932 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445377    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 13:02:18.456932 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445379    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 13:02:18.456932 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445382    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 13:02:18.457476 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445386    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 13:02:18.457476 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445389    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 13:02:18.457476 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445392    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 13:02:18.457476 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445394    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 13:02:18.457476 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445400    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 13:02:18.457476 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445403    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 13:02:18.457476 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445405    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 13:02:18.457476 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.445408    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 13:02:18.457476 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.453360    2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 13:02:18.457476 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.453374    2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 13:02:18.457476 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.453384    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 13:02:18.457476 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.453389    2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 13:02:18.457476 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453449    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 13:02:18.457476 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453455    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 13:02:18.457476 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453460    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 13:02:18.457476 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453463    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 13:02:18.457476 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453466    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 13:02:18.457476 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453469    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 13:02:18.457476 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453472    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 13:02:18.457935 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453475    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 13:02:18.457935 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453477    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 13:02:18.457935 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453480    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 13:02:18.457935 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453482    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 13:02:18.457935 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453485    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 13:02:18.457935 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453488    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 13:02:18.457935 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453491    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 13:02:18.457935 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453493    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 13:02:18.457935 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453496    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 13:02:18.457935 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453499    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 13:02:18.457935 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453501    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 13:02:18.457935 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453504    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 13:02:18.457935 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453506    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 13:02:18.457935 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453517    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 13:02:18.457935 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453520    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 13:02:18.457935 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453522    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 13:02:18.457935 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453525    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 13:02:18.457935 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453529    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 13:02:18.457935 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453531    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 13:02:18.457935 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453534    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 13:02:18.458430 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453536    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 13:02:18.458430 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453539    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 13:02:18.458430 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453542    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 13:02:18.458430 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453544    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 13:02:18.458430 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453547    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 13:02:18.458430 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453550    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 13:02:18.458430 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453553    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 13:02:18.458430 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453555    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 13:02:18.458430 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453558    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 13:02:18.458430 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453560    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 13:02:18.458430 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453563    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 13:02:18.458430 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453566    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 13:02:18.458430 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453569    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 13:02:18.458430 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453572    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 13:02:18.458430 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453574    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 13:02:18.458430 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453577    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 13:02:18.458430 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453579    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 13:02:18.458430 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453582    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 13:02:18.458430 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453584    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 13:02:18.458430 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453587    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 13:02:18.458887 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453589    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 13:02:18.458887 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453592    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 13:02:18.458887 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453595    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 13:02:18.458887 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453597    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 13:02:18.458887 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453600    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 13:02:18.458887 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453603    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 13:02:18.458887 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453607    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 13:02:18.458887 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453609    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 13:02:18.458887 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453612    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 13:02:18.458887 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453615    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 13:02:18.458887 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453618    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 13:02:18.458887 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453621    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 13:02:18.458887 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453623    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 13:02:18.458887 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453626    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 13:02:18.458887 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453628    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 13:02:18.458887 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453634    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 13:02:18.458887 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453637    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 13:02:18.458887 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453640    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 13:02:18.458887 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453642    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 13:02:18.458887 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453644    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 13:02:18.459373 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453647    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 13:02:18.459373 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453650    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 13:02:18.459373 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453652    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 13:02:18.459373 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453654    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 13:02:18.459373 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453657    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 13:02:18.459373 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453659    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 13:02:18.459373 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453662    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 13:02:18.459373 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453664    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 13:02:18.459373 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453667    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 13:02:18.459373 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453669    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 13:02:18.459373 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453672    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 13:02:18.459373 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453674    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 13:02:18.459373 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453677    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 13:02:18.459373 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453679    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 13:02:18.459373 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453682    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 13:02:18.459373 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453685    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 13:02:18.459373 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453687    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 13:02:18.459373 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453690    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 13:02:18.459373 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453693    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 13:02:18.459373 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453696    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 13:02:18.459875 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453698    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 13:02:18.459875 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453701    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 13:02:18.459875 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453704    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 13:02:18.459875 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453706    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 13:02:18.459875 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453709    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 13:02:18.459875 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453713    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 13:02:18.459875 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453717    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 13:02:18.459875 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453720    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 13:02:18.459875 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453723    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 13:02:18.459875 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453725    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 13:02:18.459875 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453728    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 13:02:18.459875 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453731    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 13:02:18.459875 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453733    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 13:02:18.459875 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453736    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 13:02:18.459875 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453739    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 13:02:18.459875 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453741    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 13:02:18.459875 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453744    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 13:02:18.459875 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453747    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 13:02:18.459875 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453750    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 13:02:18.460330 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.453779    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 13:02:18.460330 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453926    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 13:02:18.460330 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453931    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 13:02:18.460330 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453933    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 13:02:18.460330 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453936    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 13:02:18.460330 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453939    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 13:02:18.460330 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453942    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 13:02:18.460330 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453944    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 13:02:18.460330 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453947    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 13:02:18.460330 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453950    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 13:02:18.460330 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453952    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 13:02:18.460330 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453955    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 13:02:18.460330 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453957    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 13:02:18.460330 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453961    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 13:02:18.460330 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453964    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 13:02:18.460704 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453967    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 13:02:18.460704 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453969    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 13:02:18.460704 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453971    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 13:02:18.460704 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453974    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 13:02:18.460704 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453976    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 13:02:18.460704 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453979    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 13:02:18.460704 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453981    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 13:02:18.460704 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453984    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 13:02:18.460704 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453987    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 13:02:18.460704 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453989    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 13:02:18.460704 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453992    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 13:02:18.460704 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453994    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 13:02:18.460704 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453997    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 13:02:18.460704 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.453999    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 13:02:18.460704 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454002    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 13:02:18.460704 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454005    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 13:02:18.460704 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454008    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 13:02:18.460704 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454011    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 13:02:18.460704 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454013    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 13:02:18.461171 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454016    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 13:02:18.461171 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454019    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 13:02:18.461171 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454021    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 13:02:18.461171 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454024    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 13:02:18.461171 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454026    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 13:02:18.461171 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454029    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 13:02:18.461171 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454033    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 13:02:18.461171 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454036    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 13:02:18.461171 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454040    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 13:02:18.461171 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454043    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 13:02:18.461171 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454045    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 13:02:18.461171 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454048    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 13:02:18.461171 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454051    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 13:02:18.461171 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454053    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 13:02:18.461171 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454056    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 13:02:18.461171 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454058    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 13:02:18.461171 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454061    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 13:02:18.461171 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454063    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 13:02:18.461171 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454066    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 13:02:18.461626 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454069    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 13:02:18.461626 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454071    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 13:02:18.461626 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454074    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 13:02:18.461626 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454076    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 13:02:18.461626 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454079    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 13:02:18.461626 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454081    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 13:02:18.461626 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454083    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 13:02:18.461626 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454086    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 13:02:18.461626 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454088    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 13:02:18.461626 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454091    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 13:02:18.461626 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454094    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 13:02:18.461626 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454097    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 13:02:18.461626 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454100    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 13:02:18.461626 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454102    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 13:02:18.461626 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454105    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 13:02:18.461626 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454107    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 13:02:18.461626 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454110    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 13:02:18.461626 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454112    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 13:02:18.461626 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454115    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 13:02:18.461626 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454118    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 13:02:18.462097 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454120    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 13:02:18.462097 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454123    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 13:02:18.462097 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454144    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 13:02:18.462097 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454148    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 13:02:18.462097 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454151    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 13:02:18.462097 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454154    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 13:02:18.462097 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454157    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 13:02:18.462097 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454160    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 13:02:18.462097 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454163    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 13:02:18.462097 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454165    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 13:02:18.462097 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454168    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 13:02:18.462097 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454170    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 13:02:18.462097 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454173    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 13:02:18.462097 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454175    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 13:02:18.462097 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454178    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 13:02:18.462097 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454180    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 13:02:18.462097 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454183    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 13:02:18.462097 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454186    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 13:02:18.462097 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454188    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 13:02:18.462097 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454191    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 13:02:18.462585 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454193    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 13:02:18.462585 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454197    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 13:02:18.462585 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454202    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 13:02:18.462585 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454205    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 13:02:18.462585 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454208    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 13:02:18.462585 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454211    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 13:02:18.462585 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454214    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 13:02:18.462585 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454217    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 13:02:18.462585 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454219    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 13:02:18.462585 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454222    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 13:02:18.462585 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454224    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 13:02:18.462585 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454227    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 13:02:18.462585 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454229    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 13:02:18.462585 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454232    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 13:02:18.462585 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.454257    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 13:02:18.462936 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454292    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 13:02:18.462936 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454296    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 13:02:18.462936 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454299    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 13:02:18.462936 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454302    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 13:02:18.462936 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454305    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 13:02:18.462936 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454308    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 13:02:18.462936 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454310    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 13:02:18.462936 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454313    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 13:02:18.462936 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454316    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 13:02:18.462936 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454318    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 13:02:18.462936 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454321    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 13:02:18.462936 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454326    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 13:02:18.462936 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454328    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 13:02:18.462936 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454331    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 13:02:18.462936 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454334    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 13:02:18.462936 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454337    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 13:02:18.462936 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454339    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 13:02:18.462936 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454342    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 13:02:18.462936 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454346    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 13:02:18.462936 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454349    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 13:02:18.463412 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454352    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 13:02:18.463412 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454355    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 13:02:18.463412 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454358    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 13:02:18.463412 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454360    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 13:02:18.463412 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454363    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 13:02:18.463412 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454365    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 13:02:18.463412 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454368    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 13:02:18.463412 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454370    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 13:02:18.463412 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454373    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 13:02:18.463412 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454375    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 13:02:18.463412 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454378    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 13:02:18.463412 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454380    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 13:02:18.463412 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454383    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 13:02:18.463412 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454385    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 13:02:18.463412 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454388    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 13:02:18.463412 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454391    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 13:02:18.463412 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454393    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 13:02:18.463412 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454396    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 13:02:18.463412 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454399    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 13:02:18.463882 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454401    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 13:02:18.463882 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454404    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 13:02:18.463882 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454407    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 13:02:18.463882 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454409    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 13:02:18.463882 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454413    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 13:02:18.463882 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454416    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 13:02:18.463882 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454419    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 13:02:18.463882 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454421    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 13:02:18.463882 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454424    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 13:02:18.463882 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454427    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 13:02:18.463882 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454430    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 13:02:18.463882 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454432    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 13:02:18.463882 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454435    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 13:02:18.463882 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454437    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 13:02:18.463882 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454441    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 13:02:18.463882 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454443    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 13:02:18.463882 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454446    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 13:02:18.463882 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454448    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 13:02:18.463882 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454451    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 13:02:18.463882 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454454    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 13:02:18.464649 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454456    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 13:02:18.464649 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454459    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 13:02:18.464649 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454461    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 13:02:18.464649 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454464    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 13:02:18.464649 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454466    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 13:02:18.464649 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454469    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 13:02:18.464649 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454471    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 13:02:18.464649 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454474    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 13:02:18.464649 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454477    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 13:02:18.464649 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454479    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 13:02:18.464649 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454482    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 13:02:18.464649 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454486    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 13:02:18.464649 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454490    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 13:02:18.464649 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454493    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 13:02:18.464649 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454496    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 13:02:18.464649 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454499    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 13:02:18.464649 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454502    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 13:02:18.464649 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454506    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 13:02:18.464649 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454508    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 13:02:18.465375 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454511    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 13:02:18.465375 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454513    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 13:02:18.465375 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454517    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 13:02:18.465375 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454520    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 13:02:18.465375 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454523    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 13:02:18.465375 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454525    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 13:02:18.465375 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454528    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 13:02:18.465375 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454530    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 13:02:18.465375 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454533    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 13:02:18.465375 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454535    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 13:02:18.465375 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454538    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 13:02:18.465375 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454541    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 13:02:18.465375 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454543    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 13:02:18.465375 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454546    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 13:02:18.465375 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454548    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 13:02:18.465375 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454551    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 13:02:18.465375 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454553    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 13:02:18.465375 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454556    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 13:02:18.465375 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454558    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 13:02:18.465375 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454561    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 13:02:18.465895 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454563    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 13:02:18.465895 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454565    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 13:02:18.465895 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454568    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 13:02:18.465895 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454571    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 13:02:18.465895 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454574    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 13:02:18.465895 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454577    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 13:02:18.465895 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454579    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 13:02:18.465895 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:18.454582    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 13:02:18.465895 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.454635    2576 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 13:02:18.465895 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.457061    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 13:02:18.465895 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.458182    2576 server.go:1014] "Starting client certificate rotation"
Oct 30 13:02:18.465895 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.458274    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 13:02:18.465895 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.458315    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 13:02:18.465895 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.460546    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 13:02:18.465895 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.461862    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 13:02:18.475524 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.475504    2576 log.go:25] "Validated CRI v1 runtime API"
Oct 30 13:02:18.479701 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.479681    2576 log.go:25] "Validated CRI v1 image API"
Oct 30 13:02:18.480644 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.480609    2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 13:02:18.485861 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.485840    2576 fs.go:135] Filesystem UUIDs: map[3162dc0a-d86f-4d0d-89f6-a49ea6a12806:/dev/nvme0n1p4 5336019c-c3a9-4bf3-a44b-982186316a05:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Oct 30 13:02:18.485942 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.485859    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 13:02:18.491219 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.491086    2576 manager.go:217] Machine: {Timestamp:2026-10-30 13:02:18.489665429 +0000 UTC m=+0.370900640 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3089921 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22119be52f63be34f6b5f5a3d68e91 SystemUUID:ec22119b-e52f-63be-34f6-b5f5a3d68e91 BootID:cff3fcb7-0044-495e-bdaf-bbea1d4e2f7f Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:cb:02:b8:db Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:cb:02:b8:db Speed:0 Mtu:9001} {Name:ovs-system MacAddress:66:a0:e9:62:7a:be Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 13:02:18.491899 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.491888    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 13:02:18.492047 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.492034    2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 13:02:18.495067 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.495041    2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 13:02:18.495232 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.495069    2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-31.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 13:02:18.495277 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.495235    2576 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 13:02:18.495277 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.495244    2576 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 13:02:18.495331 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.495288    2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 13:02:18.495331 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.495307    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 13:02:18.518260 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.518241    2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 13:02:18.523177 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.523161    2576 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 13:02:18.523332 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.523321    2576 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 13:02:18.524189 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.524168    2576 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 13:02:18.524250 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.524199    2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 13:02:18.524250 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.524214    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 13:02:18.524250 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.524229    2576 kubelet.go:405] "Adding apiserver pod source"
Oct 30 13:02:18.524250 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.524242    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 13:02:18.524634 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.524619    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 13:02:18.527220 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.527202    2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 13:02:18.534874 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.534858    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 13:02:18.535863 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.535844    2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 13:02:18.535908 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.535887    2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 13:02:18.536095 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.536084    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 13:02:18.536157 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.536103    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 13:02:18.536157 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.536112    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 13:02:18.536157 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.536120    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 13:02:18.536157 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.536142    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 13:02:18.536263 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.536160    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 13:02:18.536726 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.536717    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 13:02:18.536754 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.536728    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 13:02:18.536754 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.536738    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 13:02:18.536754 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.536747    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 13:02:18.536830 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.536759    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 13:02:18.536830 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.536772    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 13:02:18.538240 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.538228    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 13:02:18.538288 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.538242    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 13:02:18.543078 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.543056    2576 server.go:1263] "Started kubelet"
Oct 30 13:02:18.543179 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.543149    2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 13:02:18.543594 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.543514    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 13:02:18.543819 ip-10-0-134-31 systemd[1]: Started Kubernetes Kubelet.
Oct 30 13:02:18.546680 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.546559    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 13:02:18.548188 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.548165    2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 13:02:18.550173 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.550157    2576 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 13:02:18.552702 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:18.552682    2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 13:02:18.555956 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.555935    2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 13:02:18.555956 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.555943    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 13:02:18.556198 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:18.556162    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-31.ec2.internal\" not found"
Oct 30 13:02:18.556765 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.556746    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 13:02:18.556877 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.556851    2576 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 13:02:18.556978 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.556961    2576 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 13:02:18.557079 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.557055    2576 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 13:02:18.557079 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.557065    2576 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 13:02:18.557335 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.557316    2576 factory.go:153] Registering CRI-O factory
Oct 30 13:02:18.557335 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.557337    2576 factory.go:223] Registration of the crio container factory successfully
Oct 30 13:02:18.557465 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.557388    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 13:02:18.557465 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.557396    2576 factory.go:55] Registering systemd factory
Oct 30 13:02:18.557465 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.557401    2576 factory.go:223] Registration of the systemd container factory successfully
Oct 30 13:02:18.557465 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.557444    2576 factory.go:103] Registering Raw factory
Oct 30 13:02:18.557465 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.557453    2576 manager.go:1199] Started watching for new ooms in manager
Oct 30 13:02:18.558583 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.558555    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 13:02:18.558693 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.558592    2576 manager.go:319] Starting recovery of all containers
Oct 30 13:02:18.568905 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.568883    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-31.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 13:02:18.569003 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.568941    2576 manager.go:324] Recovery completed
Oct 30 13:02:18.569835 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:18.568198    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-31.ec2.internal.18e35046145ae140  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-31.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-31.ec2.internal,},FirstTimestamp:2026-10-30 13:02:18.5430224 +0000 UTC m=+0.424257613,LastTimestamp:2026-10-30 13:02:18.5430224 +0000 UTC m=+0.424257613,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-31.ec2.internal,}"
Oct 30 13:02:18.571397 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.571379    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cn8bn"
Oct 30 13:02:18.575374 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.575357    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 13:02:18.576463 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.576447    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cn8bn"
Oct 30 13:02:18.577244 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.577223    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-31.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 13:02:18.577321 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.577256    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-31.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 13:02:18.577321 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.577271    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-31.ec2.internal" event="NodeHasSufficientPID"
Oct 30 13:02:18.577905 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.577889    2576 cpu_manager.go:226] "Starting" policy="none"
Oct 30 13:02:18.577905 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.577902    2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 13:02:18.577905 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.577903    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 13:02:18.578068 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.577918    2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 13:02:18.578647 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.578631    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-31.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 13:02:18.578706 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.578658    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-31.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 13:02:18.578706 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.578667    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-31.ec2.internal" event="NodeHasSufficientPID"
Oct 30 13:02:18.580092 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.580078    2576 policy_none.go:50] "Start"
Oct 30 13:02:18.580092 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.580094    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 13:02:18.580202 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.580105    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 13:02:18.582688 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.582678    2576 policy_none.go:44] "Start"
Oct 30 13:02:18.604580 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.604564    2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 13:02:18.638235 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.619745    2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 13:02:18.638235 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.622297    2576 manager.go:341] "Starting Device Plugin manager"
Oct 30 13:02:18.638235 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:18.622325    2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 13:02:18.638235 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.622333    2576 server.go:87] "Starting device plugin registration server"
Oct 30 13:02:18.638235 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.622554    2576 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 13:02:18.638235 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.622563    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 13:02:18.638235 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.622643    2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 13:02:18.638235 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.622742    2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 13:02:18.638235 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.622760    2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 13:02:18.638235 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:18.623297    2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 13:02:18.638235 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:18.623489    2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-31.ec2.internal\" not found"
Oct 30 13:02:18.638235 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.631083    2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 13:02:18.638235 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.631574    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 13:02:18.638235 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.632405    2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 13:02:18.638235 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.632874    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 13:02:18.638235 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.632895    2576 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 13:02:18.638235 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.632923    2576 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 13:02:18.638235 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:18.632957    2576 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 13:02:18.646501 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:18.646482    2576 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-134-31.ec2.internal\" not found" node="ip-10-0-134-31.ec2.internal"
Oct 30 13:02:18.651117 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.651095    2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 13:02:18.723241 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.723159    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 13:02:18.724165 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.724149    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-31.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 13:02:18.724272 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.724184    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-31.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 13:02:18.724272 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.724197    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-31.ec2.internal" event="NodeHasSufficientPID"
Oct 30 13:02:18.724272 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.724232    2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-31.ec2.internal"
Oct 30 13:02:18.733953 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.733934    2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-31.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-31.ec2.internal"]
Oct 30 13:02:18.734020 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.733998    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 13:02:18.734917 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.734894    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-31.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 13:02:18.735018 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.734926    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-31.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 13:02:18.735018 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.734944    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-31.ec2.internal" event="NodeHasSufficientPID"
Oct 30 13:02:18.737140 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.737119    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 13:02:18.737284 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.737269    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-31.ec2.internal"
Oct 30 13:02:18.737342 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.737308    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 13:02:18.738841 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.738032    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-31.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 13:02:18.738841 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.738060    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-31.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 13:02:18.738841 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.738073    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-31.ec2.internal" event="NodeHasSufficientPID"
Oct 30 13:02:18.738841 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.738834    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-31.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 13:02:18.739056 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.738861    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-31.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 13:02:18.739056 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.738875    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-31.ec2.internal" event="NodeHasSufficientPID"
Oct 30 13:02:18.740349 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.740328    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-31.ec2.internal"
Oct 30 13:02:18.740432 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.740356    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 13:02:18.741070 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.741055    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-31.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 13:02:18.741155 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.741083    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-31.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 13:02:18.741155 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.741095    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-31.ec2.internal" event="NodeHasSufficientPID"
Oct 30 13:02:18.745567 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.745552    2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-31.ec2.internal"
Oct 30 13:02:18.745638 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:18.745585    2576 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-31.ec2.internal\": node \"ip-10-0-134-31.ec2.internal\" not found"
Oct 30 13:02:18.756668 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.756653    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-31.ec2.internal"
Oct 30 13:02:18.776782 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.776767    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-31.ec2.internal"
Oct 30 13:02:18.779709 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.779695    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-31.ec2.internal"
Oct 30 13:02:18.857060 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.857033    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 13:02:18.858008 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.857994    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-31.ec2.internal"
Oct 30 13:02:18.858200 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.858180    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/87f9e67ec9b4ec6ee729521e7847a5a2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-31.ec2.internal\" (UID: \"87f9e67ec9b4ec6ee729521e7847a5a2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-31.ec2.internal"
Oct 30 13:02:18.858240 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.858217    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87f9e67ec9b4ec6ee729521e7847a5a2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-31.ec2.internal\" (UID: \"87f9e67ec9b4ec6ee729521e7847a5a2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-31.ec2.internal"
Oct 30 13:02:18.858273 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.858236    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5213eb0d1df39bb19f1777f20f292baf-config\") pod \"kube-apiserver-proxy-ip-10-0-134-31.ec2.internal\" (UID: \"5213eb0d1df39bb19f1777f20f292baf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-31.ec2.internal"
Oct 30 13:02:18.860273 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.860258    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 13:02:18.861975 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.861961    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 13:02:18.862019 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:18.861998    2576 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-31.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-31.ec2.internal"
Oct 30 13:02:18.886490 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.886469    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 13:02:18.886593 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:18.886503    2576 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-31.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-31.ec2.internal"
Oct 30 13:02:18.959238 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.959197    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/87f9e67ec9b4ec6ee729521e7847a5a2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-31.ec2.internal\" (UID: \"87f9e67ec9b4ec6ee729521e7847a5a2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-31.ec2.internal"
Oct 30 13:02:18.959357 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.959236    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87f9e67ec9b4ec6ee729521e7847a5a2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-31.ec2.internal\" (UID: \"87f9e67ec9b4ec6ee729521e7847a5a2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-31.ec2.internal"
Oct 30 13:02:18.959357 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.959268    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5213eb0d1df39bb19f1777f20f292baf-config\") pod \"kube-apiserver-proxy-ip-10-0-134-31.ec2.internal\" (UID: \"5213eb0d1df39bb19f1777f20f292baf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-31.ec2.internal"
Oct 30 13:02:18.959357 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.959318    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87f9e67ec9b4ec6ee729521e7847a5a2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-31.ec2.internal\" (UID: \"87f9e67ec9b4ec6ee729521e7847a5a2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-31.ec2.internal"
Oct 30 13:02:18.959477 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.959318    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/87f9e67ec9b4ec6ee729521e7847a5a2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-31.ec2.internal\" (UID: \"87f9e67ec9b4ec6ee729521e7847a5a2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-31.ec2.internal"
Oct 30 13:02:18.959477 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:18.959321    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5213eb0d1df39bb19f1777f20f292baf-config\") pod \"kube-apiserver-proxy-ip-10-0-134-31.ec2.internal\" (UID: \"5213eb0d1df39bb19f1777f20f292baf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-31.ec2.internal"
Oct 30 13:02:19.165340 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.165243    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-31.ec2.internal"
Oct 30 13:02:19.166462 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.166440    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-31.ec2.internal"
Oct 30 13:02:19.459264 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.459235    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 13:02:19.459840 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.459383    2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 13:02:19.459840 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.459416    2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 13:02:19.459840 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.459430    2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 13:02:19.524699 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.524662    2576 apiserver.go:52] "Watching apiserver"
Oct 30 13:02:19.556348 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.556319    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 13:02:19.566035 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.566012    2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 13:02:19.566351 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.566333    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-f7mvz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-31.ec2.internal","openshift-multus/multus-additional-cni-plugins-lrqnm","openshift-multus/multus-f2bqp","openshift-multus/network-metrics-daemon-mbncl","kube-system/konnectivity-agent-wpqvv","openshift-dns/node-resolver-gltng","openshift-ovn-kubernetes/ovnkube-node-67tnm","kube-system/global-pull-secret-syncer-wrfqz","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q945","openshift-network-diagnostics/network-check-target-rxd99","openshift-network-operator/iptables-alerter-5swg6","kube-system/kube-apiserver-proxy-ip-10-0-134-31.ec2.internal","openshift-cluster-node-tuning-operator/tuned-s9vk9"]
Oct 30 13:02:19.568985 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.568967    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-f7mvz"
Oct 30 13:02:19.571367 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.571345    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lrqnm"
Oct 30 13:02:19.573965 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.573946    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-f2bqp"
Oct 30 13:02:19.576005 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.575992    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbncl"
Oct 30 13:02:19.576064 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:19.576049    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mbncl" podUID="a32a253a-ecaf-4f90-af1c-27a7e53b9629"
Oct 30 13:02:19.578268 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.578255    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wpqvv"
Oct 30 13:02:19.581316 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.581293    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gltng"
Oct 30 13:02:19.582863 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.582834    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 12:57:19 +0000 UTC" deadline="2028-08-03 16:44:37.416747422 +0000 UTC"
Oct 30 13:02:19.582958 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.582863    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15435h42m17.83388764s"
Oct 30 13:02:19.583739 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.583720    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.584653 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.584625    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 13:02:19.584653 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.584639    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 13:02:19.584819 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.584738    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 13:02:19.584819 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.584751    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mxh59\""
Oct 30 13:02:19.585417 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.585400    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 13:02:19.585571 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.585490    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 13:02:19.585571 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.585514    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 13:02:19.585707 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.585660    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 13:02:19.585707 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.585672    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 13:02:19.585791 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.585709    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zbfrs\""
Oct 30 13:02:19.586042 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.586027    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrfqz"
Oct 30 13:02:19.586114 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:19.586095    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrfqz" podUID="10b774a1-1476-4f91-b6e5-d52b3f45700e"
Oct 30 13:02:19.588621 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.588605    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q945"
Oct 30 13:02:19.591507 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.591476    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxd99"
Oct 30 13:02:19.591592 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:19.591560    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rxd99" podUID="a07f5a09-920e-462f-ab6d-7a2ac53d199e"
Oct 30 13:02:19.591813 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.591794    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 13:02:19.591889 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.591869    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rrlh9\""
Oct 30 13:02:19.592261 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.592241    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 13:02:19.592383 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.592367    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 13:02:19.592451 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.592367    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-r278v\""
Oct 30 13:02:19.594045 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.594028    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5swg6"
Oct 30 13:02:19.596583 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.596564    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:02:19.601294 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.601272    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 13:02:19.601390 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.601332    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-x6852"
Oct 30 13:02:19.607339 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.607319    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-x6852"
Oct 30 13:02:19.658165 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.658145    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 13:02:19.662680 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.662651    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-host-run-ovn-kubernetes\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.662795 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.662693    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a323e312-dd91-486f-985d-92b9ab8cca04-socket-dir\") pod \"aws-ebs-csi-driver-node-9q945\" (UID: \"a323e312-dd91-486f-985d-92b9ab8cca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q945"
Oct 30 13:02:19.662795 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.662725    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5a16e7b2-e45d-480a-a7bc-d914640b9867-hosts-file\") pod \"node-resolver-gltng\" (UID: \"5a16e7b2-e45d-480a-a7bc-d914640b9867\") " pod="openshift-dns/node-resolver-gltng"
Oct 30 13:02:19.662880 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.662799    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a16e7b2-e45d-480a-a7bc-d914640b9867-tmp-dir\") pod \"node-resolver-gltng\" (UID: \"5a16e7b2-e45d-480a-a7bc-d914640b9867\") " pod="openshift-dns/node-resolver-gltng"
Oct 30 13:02:19.662880 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.662862    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-lib-modules\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:02:19.662979 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.662927    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xws2p\" (UniqueName: \"kubernetes.io/projected/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-kube-api-access-xws2p\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.663036 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.662973    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/087058f5-e286-4ac7-8ed8-a280da332878-serviceca\") pod \"node-ca-f7mvz\" (UID: \"087058f5-e286-4ac7-8ed8-a280da332878\") " pod="openshift-image-registry/node-ca-f7mvz"
Oct 30 13:02:19.663036 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.663016    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-multus-cni-dir\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:02:19.663145 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.663045    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-host-var-lib-cni-bin\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:02:19.663145 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.663081    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-host-run-multus-certs\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:02:19.663145 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.663117    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/beb2b7e4-b43d-42fc-9e8a-ecdfa6994205-konnectivity-ca\") pod \"konnectivity-agent-wpqvv\" (UID: \"beb2b7e4-b43d-42fc-9e8a-ecdfa6994205\") " pod="kube-system/konnectivity-agent-wpqvv"
Oct 30 13:02:19.663327 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.663170    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-etc-sysctl-conf\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:02:19.663327 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.663201    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/183f23e7-ae80-445b-960c-fff9b410ba5b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lrqnm\" (UID: \"183f23e7-ae80-445b-960c-fff9b410ba5b\") " pod="openshift-multus/multus-additional-cni-plugins-lrqnm"
Oct 30 13:02:19.663327 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.663228    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-host-kubelet\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.663327 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.663255    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-ovn-node-metrics-cert\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.663327 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.663318    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/de69ec51-fd3a-4bc4-ac17-ff02bc582905-cni-binary-copy\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:02:19.663556 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.663346    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-multus-socket-dir-parent\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:02:19.663556 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.663376    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz5ls\" (UniqueName: \"kubernetes.io/projected/a07f5a09-920e-462f-ab6d-7a2ac53d199e-kube-api-access-gz5ls\") pod \"network-check-target-rxd99\" (UID: \"a07f5a09-920e-462f-ab6d-7a2ac53d199e\") " pod="openshift-network-diagnostics/network-check-target-rxd99"
Oct 30 13:02:19.663556 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.663408    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-etc-sysconfig\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:02:19.663556 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.663468    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-sys\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:02:19.663556 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.663515    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb526\" (UniqueName: \"kubernetes.io/projected/f1fb4abc-b161-461c-b8ef-bd1212010510-kube-api-access-zb526\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:02:19.663556 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.663544    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-host-cni-netd\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.663802 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.663597    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-ovnkube-script-lib\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.663802 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.663652    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whwnz\" (UniqueName: \"kubernetes.io/projected/a323e312-dd91-486f-985d-92b9ab8cca04-kube-api-access-whwnz\") pod \"aws-ebs-csi-driver-node-9q945\" (UID: \"a323e312-dd91-486f-985d-92b9ab8cca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q945"
Oct 30 13:02:19.663802 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.663674    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-hostroot\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:02:19.663802 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.663690    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1fb4abc-b161-461c-b8ef-bd1212010510-tmp\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:02:19.663802 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.663730    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/183f23e7-ae80-445b-960c-fff9b410ba5b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lrqnm\" (UID: \"183f23e7-ae80-445b-960c-fff9b410ba5b\") " pod="openshift-multus/multus-additional-cni-plugins-lrqnm"
Oct 30 13:02:19.663802 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.663762    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-host-run-netns\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.664024 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.663821    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-host\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:02:19.664024 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.663838    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a32a253a-ecaf-4f90-af1c-27a7e53b9629-metrics-certs\") pod \"network-metrics-daemon-mbncl\" (UID: \"a32a253a-ecaf-4f90-af1c-27a7e53b9629\") " pod="openshift-multus/network-metrics-daemon-mbncl"
Oct 30 13:02:19.664024 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.663855    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-systemd-units\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.664024 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.663871    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-host-slash\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.664024 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.663887    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-run-systemd\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.664024 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.663985    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-log-socket\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.664024 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.664003    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6f7b\" (UniqueName: \"kubernetes.io/projected/de69ec51-fd3a-4bc4-ac17-ff02bc582905-kube-api-access-v6f7b\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:02:19.664024 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.664018    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjgw5\" (UniqueName: \"kubernetes.io/projected/183f23e7-ae80-445b-960c-fff9b410ba5b-kube-api-access-hjgw5\") pod \"multus-additional-cni-plugins-lrqnm\" (UID: \"183f23e7-ae80-445b-960c-fff9b410ba5b\") " pod="openshift-multus/multus-additional-cni-plugins-lrqnm"
Oct 30 13:02:19.664417 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.664044    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/10b774a1-1476-4f91-b6e5-d52b3f45700e-original-pull-secret\") pod \"global-pull-secret-syncer-wrfqz\" (UID: \"10b774a1-1476-4f91-b6e5-d52b3f45700e\") " pod="kube-system/global-pull-secret-syncer-wrfqz"
Oct 30 13:02:19.664417 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.664059    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/087058f5-e286-4ac7-8ed8-a280da332878-host\") pod \"node-ca-f7mvz\" (UID: \"087058f5-e286-4ac7-8ed8-a280da332878\") " pod="openshift-image-registry/node-ca-f7mvz"
Oct 30 13:02:19.664417 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.664097    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-system-cni-dir\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:02:19.664417 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.664172    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-host-run-netns\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:02:19.664417 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.664227    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-host-var-lib-kubelet\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:02:19.664417 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.664281    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/beb2b7e4-b43d-42fc-9e8a-ecdfa6994205-agent-certs\") pod \"konnectivity-agent-wpqvv\" (UID: \"beb2b7e4-b43d-42fc-9e8a-ecdfa6994205\") " pod="kube-system/konnectivity-agent-wpqvv"
Oct 30 13:02:19.664417 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.664330    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/183f23e7-ae80-445b-960c-fff9b410ba5b-system-cni-dir\") pod \"multus-additional-cni-plugins-lrqnm\" (UID: \"183f23e7-ae80-445b-960c-fff9b410ba5b\") " pod="openshift-multus/multus-additional-cni-plugins-lrqnm"
Oct 30 13:02:19.664417 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.664380    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/183f23e7-ae80-445b-960c-fff9b410ba5b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lrqnm\" (UID: \"183f23e7-ae80-445b-960c-fff9b410ba5b\") " pod="openshift-multus/multus-additional-cni-plugins-lrqnm"
Oct 30 13:02:19.664759 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.664423    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-etc-openvswitch\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.664759 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.664460    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-run-ovn\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.664759 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.664487    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/10b774a1-1476-4f91-b6e5-d52b3f45700e-dbus\") pod \"global-pull-secret-syncer-wrfqz\" (UID: \"10b774a1-1476-4f91-b6e5-d52b3f45700e\") " pod="kube-system/global-pull-secret-syncer-wrfqz"
Oct 30 13:02:19.664759 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.664520    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/183f23e7-ae80-445b-960c-fff9b410ba5b-cni-binary-copy\") pod \"multus-additional-cni-plugins-lrqnm\" (UID: \"183f23e7-ae80-445b-960c-fff9b410ba5b\") " pod="openshift-multus/multus-additional-cni-plugins-lrqnm"
Oct 30 13:02:19.664759 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.664563    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-ovnkube-config\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.664759 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.664650    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-cnibin\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:02:19.664759 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.664698    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-multus-conf-dir\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:02:19.664759 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.664727    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-etc-systemd\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:02:19.665140 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.664780    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsfql\" (UniqueName: \"kubernetes.io/projected/a32a253a-ecaf-4f90-af1c-27a7e53b9629-kube-api-access-qsfql\") pod \"network-metrics-daemon-mbncl\" (UID: \"a32a253a-ecaf-4f90-af1c-27a7e53b9629\") " pod="openshift-multus/network-metrics-daemon-mbncl"
Oct 30 13:02:19.665140 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.664809    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-node-log\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.665140 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.664837    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/de69ec51-fd3a-4bc4-ac17-ff02bc582905-multus-daemon-config\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:02:19.665140 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.664865    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f1fb4abc-b161-461c-b8ef-bd1212010510-etc-tuned\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:02:19.665140 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.664922    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0c8e72ed-d54d-4e6a-bf66-65a16430351d-host-slash\") pod \"iptables-alerter-5swg6\" (UID: \"0c8e72ed-d54d-4e6a-bf66-65a16430351d\") " pod="openshift-network-operator/iptables-alerter-5swg6"
Oct 30 13:02:19.665140 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.664950    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-var-lib-kubelet\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:02:19.665140 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.664976    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a323e312-dd91-486f-985d-92b9ab8cca04-registration-dir\") pod \"aws-ebs-csi-driver-node-9q945\" (UID: \"a323e312-dd91-486f-985d-92b9ab8cca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q945"
Oct 30 13:02:19.665140 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.665003    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a323e312-dd91-486f-985d-92b9ab8cca04-sys-fs\") pod \"aws-ebs-csi-driver-node-9q945\" (UID: \"a323e312-dd91-486f-985d-92b9ab8cca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q945"
Oct 30 13:02:19.665140 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.665035    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-host-run-k8s-cni-cncf-io\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:02:19.665140 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.665066    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-etc-kubernetes\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:02:19.665140 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.665099    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a323e312-dd91-486f-985d-92b9ab8cca04-device-dir\") pod \"aws-ebs-csi-driver-node-9q945\" (UID: \"a323e312-dd91-486f-985d-92b9ab8cca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q945"
Oct 30 13:02:19.665751 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.665198    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0c8e72ed-d54d-4e6a-bf66-65a16430351d-iptables-alerter-script\") pod \"iptables-alerter-5swg6\" (UID: \"0c8e72ed-d54d-4e6a-bf66-65a16430351d\") " pod="openshift-network-operator/iptables-alerter-5swg6"
Oct 30 13:02:19.665751 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.665228    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-run\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:02:19.665751 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.665266    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-host-cni-bin\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.665751 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.665293    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-env-overrides\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.665751 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.665322    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/10b774a1-1476-4f91-b6e5-d52b3f45700e-kubelet-config\") pod \"global-pull-secret-syncer-wrfqz\" (UID: \"10b774a1-1476-4f91-b6e5-d52b3f45700e\") " pod="kube-system/global-pull-secret-syncer-wrfqz"
Oct 30 13:02:19.665751 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.665350    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-host-var-lib-cni-multus\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:02:19.665751 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.665389    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-etc-kubernetes\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:02:19.665751 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.665416    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a323e312-dd91-486f-985d-92b9ab8cca04-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9q945\" (UID: \"a323e312-dd91-486f-985d-92b9ab8cca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q945"
Oct 30 13:02:19.665751 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.665447    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a323e312-dd91-486f-985d-92b9ab8cca04-etc-selinux\") pod \"aws-ebs-csi-driver-node-9q945\" (UID: \"a323e312-dd91-486f-985d-92b9ab8cca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q945"
Oct 30 13:02:19.665751 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.665474    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-etc-modprobe-d\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:02:19.665751 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.665503    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-etc-sysctl-d\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:02:19.665751 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.665531    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/183f23e7-ae80-445b-960c-fff9b410ba5b-cnibin\") pod \"multus-additional-cni-plugins-lrqnm\" (UID: \"183f23e7-ae80-445b-960c-fff9b410ba5b\") " pod="openshift-multus/multus-additional-cni-plugins-lrqnm"
Oct 30 13:02:19.665751 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.665559    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-var-lib-openvswitch\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.665751 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.665590    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-run-openvswitch\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.665751 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.665621    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgqhj\" (UniqueName: \"kubernetes.io/projected/0c8e72ed-d54d-4e6a-bf66-65a16430351d-kube-api-access-dgqhj\") pod \"iptables-alerter-5swg6\" (UID: \"0c8e72ed-d54d-4e6a-bf66-65a16430351d\") " pod="openshift-network-operator/iptables-alerter-5swg6"
Oct 30 13:02:19.665751 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.665752    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.666524 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.665799    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gdf9\" (UniqueName: \"kubernetes.io/projected/087058f5-e286-4ac7-8ed8-a280da332878-kube-api-access-6gdf9\") pod \"node-ca-f7mvz\" (UID: \"087058f5-e286-4ac7-8ed8-a280da332878\") " pod="openshift-image-registry/node-ca-f7mvz"
Oct 30 13:02:19.666524 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.665834    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9m9z\" (UniqueName: \"kubernetes.io/projected/5a16e7b2-e45d-480a-a7bc-d914640b9867-kube-api-access-z9m9z\") pod \"node-resolver-gltng\" (UID: \"5a16e7b2-e45d-480a-a7bc-d914640b9867\") " pod="openshift-dns/node-resolver-gltng"
Oct 30 13:02:19.692917 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.692885    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 13:02:19.692917 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.692903    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 13:02:19.692917 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.692885    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 13:02:19.693247 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.693073    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 13:02:19.693247 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.693118    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 13:02:19.693247 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.693150    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 13:02:19.693409 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.693323    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-prxcs\""
Oct 30 13:02:19.693409 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.693362    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 13:02:19.693409 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.693364    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vz5h4\""
Oct 30 13:02:19.693409 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.693390    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8w9b7\""
Oct 30 13:02:19.693409 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.693407    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 13:02:19.693606 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.693417    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 13:02:19.693606 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.693422    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 13:02:19.693606 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.693411    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 13:02:19.693606 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.693447    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 13:02:19.693606 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.693483    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 13:02:19.702538 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.702517    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-m8jrt\""
Oct 30 13:02:19.702538 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.702523    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 13:02:19.702674 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.702543    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 13:02:19.705247 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.705227    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-sxrfm\""
Oct 30 13:02:19.705369 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.705233    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 13:02:19.766790 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.766709    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-etc-sysctl-conf\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:02:19.766790 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.766758    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/183f23e7-ae80-445b-960c-fff9b410ba5b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lrqnm\" (UID: \"183f23e7-ae80-445b-960c-fff9b410ba5b\") " pod="openshift-multus/multus-additional-cni-plugins-lrqnm"
Oct 30 13:02:19.766790 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.766776    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-host-kubelet\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.766790 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.766795    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-ovn-node-metrics-cert\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.767117 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.766821    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/de69ec51-fd3a-4bc4-ac17-ff02bc582905-cni-binary-copy\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:02:19.767117 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.766850    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-multus-socket-dir-parent\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:02:19.767117 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.766859    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-host-kubelet\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.767117 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.766881    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz5ls\" (UniqueName: \"kubernetes.io/projected/a07f5a09-920e-462f-ab6d-7a2ac53d199e-kube-api-access-gz5ls\") pod \"network-check-target-rxd99\" (UID: \"a07f5a09-920e-462f-ab6d-7a2ac53d199e\") " pod="openshift-network-diagnostics/network-check-target-rxd99"
Oct 30 13:02:19.767117 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.766957    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-etc-sysctl-conf\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:02:19.767117 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.766972    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-etc-sysconfig\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:02:19.767117 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.767022    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-sys\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:02:19.767117 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.767082    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-multus-socket-dir-parent\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:02:19.767494 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.767215    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb526\" (UniqueName: \"kubernetes.io/projected/f1fb4abc-b161-461c-b8ef-bd1212010510-kube-api-access-zb526\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:02:19.767613 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.767260    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-sys\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:02:19.767800 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.767751    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-host-cni-netd\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.767896 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.767711    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-host-cni-netd\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.767955 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.767901    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/183f23e7-ae80-445b-960c-fff9b410ba5b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lrqnm\" (UID: \"183f23e7-ae80-445b-960c-fff9b410ba5b\") " pod="openshift-multus/multus-additional-cni-plugins-lrqnm"
Oct 30 13:02:19.767955 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.767936    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-ovnkube-script-lib\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.768060 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.767962    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/de69ec51-fd3a-4bc4-ac17-ff02bc582905-cni-binary-copy\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:02:19.768060 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.767973    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whwnz\" (UniqueName: \"kubernetes.io/projected/a323e312-dd91-486f-985d-92b9ab8cca04-kube-api-access-whwnz\") pod \"aws-ebs-csi-driver-node-9q945\" (UID: \"a323e312-dd91-486f-985d-92b9ab8cca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q945"
Oct 30 13:02:19.768060 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.767962    2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 13:02:19.768060 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.767209    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-etc-sysconfig\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:02:19.768060 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.768017    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-hostroot\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:02:19.768338 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.768103    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1fb4abc-b161-461c-b8ef-bd1212010510-tmp\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:02:19.768338 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.768191    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-hostroot\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:02:19.770902 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.770868    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/183f23e7-ae80-445b-960c-fff9b410ba5b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lrqnm\" (UID: \"183f23e7-ae80-445b-960c-fff9b410ba5b\") " pod="openshift-multus/multus-additional-cni-plugins-lrqnm"
Oct 30 13:02:19.771046 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.771019    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-ovnkube-script-lib\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.771109 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.771056    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-host-run-netns\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.771175 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.771102    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/183f23e7-ae80-445b-960c-fff9b410ba5b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lrqnm\" (UID: \"183f23e7-ae80-445b-960c-fff9b410ba5b\") " pod="openshift-multus/multus-additional-cni-plugins-lrqnm"
Oct 30 13:02:19.771175 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.771098    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-host\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:02:19.771318 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.771179    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a32a253a-ecaf-4f90-af1c-27a7e53b9629-metrics-certs\") pod \"network-metrics-daemon-mbncl\" (UID: \"a32a253a-ecaf-4f90-af1c-27a7e53b9629\") " pod="openshift-multus/network-metrics-daemon-mbncl"
Oct 30 13:02:19.771318 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.771292    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-systemd-units\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.771318 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:19.771310    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 13:02:19.771472 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.771335    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-host-slash\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.771472 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.771360    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-host\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:02:19.771472 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.771378    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-run-systemd\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.771472 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.771433    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-log-socket\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.771472 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.771443    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-systemd-units\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.771472 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:19.771468    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a32a253a-ecaf-4f90-af1c-27a7e53b9629-metrics-certs podName:a32a253a-ecaf-4f90-af1c-27a7e53b9629 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:20.271427082 +0000 UTC m=+2.152662298 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a32a253a-ecaf-4f90-af1c-27a7e53b9629-metrics-certs") pod "network-metrics-daemon-mbncl" (UID: "a32a253a-ecaf-4f90-af1c-27a7e53b9629") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 13:02:19.771771 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.771500    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-host-slash\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.771771 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.771505    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-log-socket\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.771771 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.771519    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6f7b\" (UniqueName: \"kubernetes.io/projected/de69ec51-fd3a-4bc4-ac17-ff02bc582905-kube-api-access-v6f7b\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:02:19.771771 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.771557    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjgw5\" (UniqueName: \"kubernetes.io/projected/183f23e7-ae80-445b-960c-fff9b410ba5b-kube-api-access-hjgw5\") pod \"multus-additional-cni-plugins-lrqnm\" (UID: \"183f23e7-ae80-445b-960c-fff9b410ba5b\") " pod="openshift-multus/multus-additional-cni-plugins-lrqnm"
Oct 30 13:02:19.771771 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.771561    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-run-systemd\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.771771 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.771595    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/10b774a1-1476-4f91-b6e5-d52b3f45700e-original-pull-secret\") pod \"global-pull-secret-syncer-wrfqz\" (UID: \"10b774a1-1476-4f91-b6e5-d52b3f45700e\") " pod="kube-system/global-pull-secret-syncer-wrfqz"
Oct 30 13:02:19.771771 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.771638    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-host-run-netns\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.771771 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.771666    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/087058f5-e286-4ac7-8ed8-a280da332878-host\") pod \"node-ca-f7mvz\" (UID: \"087058f5-e286-4ac7-8ed8-a280da332878\") " pod="openshift-image-registry/node-ca-f7mvz"
Oct 30 13:02:19.771771 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.771714    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-system-cni-dir\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:02:19.771771 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.771770    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-system-cni-dir\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:02:19.772283 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.771771    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-host-run-netns\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:02:19.772283 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.771811    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-host-run-netns\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:02:19.772283 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:19.771832    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 13:02:19.772283 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.771841    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-host-var-lib-kubelet\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:02:19.772283 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.771886    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-host-var-lib-kubelet\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:02:19.772283 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.771824    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/087058f5-e286-4ac7-8ed8-a280da332878-host\") pod \"node-ca-f7mvz\" (UID: \"087058f5-e286-4ac7-8ed8-a280da332878\") " pod="openshift-image-registry/node-ca-f7mvz"
Oct 30 13:02:19.772283 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.771914    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/beb2b7e4-b43d-42fc-9e8a-ecdfa6994205-agent-certs\") pod \"konnectivity-agent-wpqvv\" (UID: \"beb2b7e4-b43d-42fc-9e8a-ecdfa6994205\") " pod="kube-system/konnectivity-agent-wpqvv"
Oct 30 13:02:19.772283 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.772008    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/183f23e7-ae80-445b-960c-fff9b410ba5b-system-cni-dir\") pod \"multus-additional-cni-plugins-lrqnm\" (UID: \"183f23e7-ae80-445b-960c-fff9b410ba5b\") " pod="openshift-multus/multus-additional-cni-plugins-lrqnm"
Oct 30 13:02:19.772283 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.772016    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/183f23e7-ae80-445b-960c-fff9b410ba5b-system-cni-dir\") pod \"multus-additional-cni-plugins-lrqnm\" (UID: \"183f23e7-ae80-445b-960c-fff9b410ba5b\") " pod="openshift-multus/multus-additional-cni-plugins-lrqnm"
Oct 30 13:02:19.772283 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.772058    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/183f23e7-ae80-445b-960c-fff9b410ba5b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lrqnm\" (UID: \"183f23e7-ae80-445b-960c-fff9b410ba5b\") " pod="openshift-multus/multus-additional-cni-plugins-lrqnm"
Oct 30 13:02:19.772283 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.772117    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-etc-openvswitch\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.772283 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.772178    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-run-ovn\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.772283 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.772250    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/10b774a1-1476-4f91-b6e5-d52b3f45700e-dbus\") pod \"global-pull-secret-syncer-wrfqz\" (UID: \"10b774a1-1476-4f91-b6e5-d52b3f45700e\") " pod="kube-system/global-pull-secret-syncer-wrfqz"
Oct 30 13:02:19.772756 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.772322    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/183f23e7-ae80-445b-960c-fff9b410ba5b-cni-binary-copy\") pod \"multus-additional-cni-plugins-lrqnm\" (UID: \"183f23e7-ae80-445b-960c-fff9b410ba5b\") " pod="openshift-multus/multus-additional-cni-plugins-lrqnm"
Oct 30 13:02:19.772756 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.772345    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-etc-openvswitch\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.772756 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.772367    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-ovnkube-config\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.773502 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.772406    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-cnibin\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:02:19.773502 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:19.773128    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10b774a1-1476-4f91-b6e5-d52b3f45700e-original-pull-secret podName:10b774a1-1476-4f91-b6e5-d52b3f45700e nodeName:}" failed. No retries permitted until 2026-10-30 13:02:20.273102412 +0000 UTC m=+2.154337631 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/10b774a1-1476-4f91-b6e5-d52b3f45700e-original-pull-secret") pod "global-pull-secret-syncer-wrfqz" (UID: "10b774a1-1476-4f91-b6e5-d52b3f45700e") : object "kube-system"/"original-pull-secret" not registered
Oct 30 13:02:19.773502 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.773228    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-run-ovn\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.773502 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.773445    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-cnibin\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:02:19.773721 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.773501    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/10b774a1-1476-4f91-b6e5-d52b3f45700e-dbus\") pod \"global-pull-secret-syncer-wrfqz\" (UID: \"10b774a1-1476-4f91-b6e5-d52b3f45700e\") " pod="kube-system/global-pull-secret-syncer-wrfqz"
Oct 30 13:02:19.773721 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.773566    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-multus-conf-dir\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:02:19.773721 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.773599    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-etc-systemd\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:02:19.773721 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.773632    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsfql\" (UniqueName: \"kubernetes.io/projected/a32a253a-ecaf-4f90-af1c-27a7e53b9629-kube-api-access-qsfql\") pod \"network-metrics-daemon-mbncl\" (UID: \"a32a253a-ecaf-4f90-af1c-27a7e53b9629\") " pod="openshift-multus/network-metrics-daemon-mbncl"
Oct 30 13:02:19.773721 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.773631    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-multus-conf-dir\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:02:19.773721 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.773657    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-node-log\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.773721 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.773666    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-etc-systemd\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:02:19.773721 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.773685    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/de69ec51-fd3a-4bc4-ac17-ff02bc582905-multus-daemon-config\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:02:19.773721 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.773720    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-node-log\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.774113 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.773726    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f1fb4abc-b161-461c-b8ef-bd1212010510-etc-tuned\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:02:19.774113 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.773753    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0c8e72ed-d54d-4e6a-bf66-65a16430351d-host-slash\") pod \"iptables-alerter-5swg6\" (UID: \"0c8e72ed-d54d-4e6a-bf66-65a16430351d\") " pod="openshift-network-operator/iptables-alerter-5swg6"
Oct 30 13:02:19.774113 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.773780    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-var-lib-kubelet\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:02:19.774113 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.773806    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a323e312-dd91-486f-985d-92b9ab8cca04-registration-dir\") pod \"aws-ebs-csi-driver-node-9q945\" (UID: \"a323e312-dd91-486f-985d-92b9ab8cca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q945"
Oct 30 13:02:19.774113 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.773863    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0c8e72ed-d54d-4e6a-bf66-65a16430351d-host-slash\") pod \"iptables-alerter-5swg6\" (UID: \"0c8e72ed-d54d-4e6a-bf66-65a16430351d\") " pod="openshift-network-operator/iptables-alerter-5swg6"
Oct 30 13:02:19.774113 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.773874    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a323e312-dd91-486f-985d-92b9ab8cca04-registration-dir\") pod \"aws-ebs-csi-driver-node-9q945\" (UID: \"a323e312-dd91-486f-985d-92b9ab8cca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q945"
Oct 30 13:02:19.774113 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.773907    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a323e312-dd91-486f-985d-92b9ab8cca04-sys-fs\") pod \"aws-ebs-csi-driver-node-9q945\" (UID: \"a323e312-dd91-486f-985d-92b9ab8cca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q945"
Oct 30 13:02:19.774113 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.773909    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-var-lib-kubelet\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:02:19.774113 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.773942    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-host-run-k8s-cni-cncf-io\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:02:19.774113 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.773954    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a323e312-dd91-486f-985d-92b9ab8cca04-sys-fs\") pod \"aws-ebs-csi-driver-node-9q945\" (UID: \"a323e312-dd91-486f-985d-92b9ab8cca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q945"
Oct 30 13:02:19.774113 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.773985    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-host-run-k8s-cni-cncf-io\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:02:19.774113 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.774018    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-etc-kubernetes\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:02:19.774113 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.774041    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-etc-kubernetes\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:02:19.774113 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.774053    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a323e312-dd91-486f-985d-92b9ab8cca04-device-dir\") pod \"aws-ebs-csi-driver-node-9q945\" (UID: \"a323e312-dd91-486f-985d-92b9ab8cca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q945"
Oct 30 13:02:19.774113 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.774099    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0c8e72ed-d54d-4e6a-bf66-65a16430351d-iptables-alerter-script\") pod \"iptables-alerter-5swg6\" (UID: \"0c8e72ed-d54d-4e6a-bf66-65a16430351d\") " pod="openshift-network-operator/iptables-alerter-5swg6"
Oct 30 13:02:19.774895 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.774125    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-run\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:02:19.774895 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.774163    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-host-cni-bin\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.774895 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.774193    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-env-overrides\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.774895 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.774197    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a323e312-dd91-486f-985d-92b9ab8cca04-device-dir\") pod \"aws-ebs-csi-driver-node-9q945\" (UID: \"a323e312-dd91-486f-985d-92b9ab8cca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q945"
Oct 30 13:02:19.774895 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.774218    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/10b774a1-1476-4f91-b6e5-d52b3f45700e-kubelet-config\") pod \"global-pull-secret-syncer-wrfqz\" (UID: \"10b774a1-1476-4f91-b6e5-d52b3f45700e\") " pod="kube-system/global-pull-secret-syncer-wrfqz"
Oct 30 13:02:19.774895 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.774253    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-host-cni-bin\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.774895 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.774246    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-host-var-lib-cni-multus\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:02:19.774895 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.774290    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-etc-kubernetes\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:02:19.774895 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.774306    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-run\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:02:19.774895 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.774317    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a323e312-dd91-486f-985d-92b9ab8cca04-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9q945\" (UID: \"a323e312-dd91-486f-985d-92b9ab8cca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q945"
Oct 30 13:02:19.774895 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.774342    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-ovnkube-config\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.774895 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.774437    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a323e312-dd91-486f-985d-92b9ab8cca04-etc-selinux\") pod \"aws-ebs-csi-driver-node-9q945\" (UID: \"a323e312-dd91-486f-985d-92b9ab8cca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q945"
Oct 30 13:02:19.774895 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.774344    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a323e312-dd91-486f-985d-92b9ab8cca04-etc-selinux\") pod \"aws-ebs-csi-driver-node-9q945\" (UID: \"a323e312-dd91-486f-985d-92b9ab8cca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q945"
Oct 30 13:02:19.774895 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.774480    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-etc-modprobe-d\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:02:19.774895 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.774506    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-etc-sysctl-d\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:02:19.774895 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.774529    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/183f23e7-ae80-445b-960c-fff9b410ba5b-cnibin\") pod \"multus-additional-cni-plugins-lrqnm\" (UID: \"183f23e7-ae80-445b-960c-fff9b410ba5b\") " pod="openshift-multus/multus-additional-cni-plugins-lrqnm"
Oct 30 13:02:19.774895 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.774554    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-var-lib-openvswitch\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.775657 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.774577    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-run-openvswitch\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.775657 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.774599    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgqhj\" (UniqueName: \"kubernetes.io/projected/0c8e72ed-d54d-4e6a-bf66-65a16430351d-kube-api-access-dgqhj\") pod \"iptables-alerter-5swg6\" (UID: \"0c8e72ed-d54d-4e6a-bf66-65a16430351d\") " pod="openshift-network-operator/iptables-alerter-5swg6"
Oct 30 13:02:19.775657 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.774636    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0c8e72ed-d54d-4e6a-bf66-65a16430351d-iptables-alerter-script\") pod \"iptables-alerter-5swg6\" (UID: \"0c8e72ed-d54d-4e6a-bf66-65a16430351d\") " pod="openshift-network-operator/iptables-alerter-5swg6"
Oct 30 13:02:19.775657 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.774653    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.775657 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.774684    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-host-var-lib-cni-multus\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:02:19.775657 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.774687    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gdf9\" (UniqueName: \"kubernetes.io/projected/087058f5-e286-4ac7-8ed8-a280da332878-kube-api-access-6gdf9\") pod \"node-ca-f7mvz\" (UID: \"087058f5-e286-4ac7-8ed8-a280da332878\") " pod="openshift-image-registry/node-ca-f7mvz"
Oct 30 13:02:19.775657 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.774714    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9m9z\" (UniqueName: \"kubernetes.io/projected/5a16e7b2-e45d-480a-a7bc-d914640b9867-kube-api-access-z9m9z\") pod \"node-resolver-gltng\" (UID: \"5a16e7b2-e45d-480a-a7bc-d914640b9867\") " pod="openshift-dns/node-resolver-gltng"
Oct 30 13:02:19.775657 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.774720    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/10b774a1-1476-4f91-b6e5-d52b3f45700e-kubelet-config\") pod \"global-pull-secret-syncer-wrfqz\" (UID: \"10b774a1-1476-4f91-b6e5-d52b3f45700e\") " pod="kube-system/global-pull-secret-syncer-wrfqz"
Oct 30 13:02:19.775657 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.774738    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-host-run-ovn-kubernetes\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.775657 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.774749    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-etc-kubernetes\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:02:19.775657 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.774758    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-env-overrides\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.775657 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.774775    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-run-openvswitch\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.775657 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.774764    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a323e312-dd91-486f-985d-92b9ab8cca04-socket-dir\") pod \"aws-ebs-csi-driver-node-9q945\" (UID: \"a323e312-dd91-486f-985d-92b9ab8cca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q945"
Oct 30 13:02:19.775657 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.774811    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/beb2b7e4-b43d-42fc-9e8a-ecdfa6994205-agent-certs\") pod \"konnectivity-agent-wpqvv\" (UID: \"beb2b7e4-b43d-42fc-9e8a-ecdfa6994205\") " pod="kube-system/konnectivity-agent-wpqvv"
Oct 30 13:02:19.775657 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.774833    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-etc-modprobe-d\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:02:19.775657 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.774861    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-var-lib-openvswitch\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.775657 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.774890    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.776137 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.774894    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5a16e7b2-e45d-480a-a7bc-d914640b9867-hosts-file\") pod \"node-resolver-gltng\" (UID: \"5a16e7b2-e45d-480a-a7bc-d914640b9867\") " pod="openshift-dns/node-resolver-gltng"
Oct 30 13:02:19.776137 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.774929    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a16e7b2-e45d-480a-a7bc-d914640b9867-tmp-dir\") pod \"node-resolver-gltng\" (UID: \"5a16e7b2-e45d-480a-a7bc-d914640b9867\") " pod="openshift-dns/node-resolver-gltng"
Oct 30 13:02:19.776137 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.774935    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5a16e7b2-e45d-480a-a7bc-d914640b9867-hosts-file\") pod \"node-resolver-gltng\" (UID: \"5a16e7b2-e45d-480a-a7bc-d914640b9867\") " pod="openshift-dns/node-resolver-gltng"
Oct 30 13:02:19.776137 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.774956    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-lib-modules\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:02:19.776137 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.774986    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/183f23e7-ae80-445b-960c-fff9b410ba5b-cnibin\") pod \"multus-additional-cni-plugins-lrqnm\" (UID: \"183f23e7-ae80-445b-960c-fff9b410ba5b\") " pod="openshift-multus/multus-additional-cni-plugins-lrqnm"
Oct 30 13:02:19.776137 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.774990    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xws2p\" (UniqueName: \"kubernetes.io/projected/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-kube-api-access-xws2p\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.776137 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.774902    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a323e312-dd91-486f-985d-92b9ab8cca04-socket-dir\") pod \"aws-ebs-csi-driver-node-9q945\" (UID: \"a323e312-dd91-486f-985d-92b9ab8cca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q945"
Oct 30 13:02:19.776137 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.775063    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-lib-modules\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:02:19.776137 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.775103    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a323e312-dd91-486f-985d-92b9ab8cca04-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9q945\" (UID: \"a323e312-dd91-486f-985d-92b9ab8cca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q945"
Oct 30 13:02:19.776137 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.775120    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-etc-sysctl-d\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:02:19.776137 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.775144    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-host-run-ovn-kubernetes\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.776137 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.775165    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/087058f5-e286-4ac7-8ed8-a280da332878-serviceca\") pod \"node-ca-f7mvz\" (UID: \"087058f5-e286-4ac7-8ed8-a280da332878\") " pod="openshift-image-registry/node-ca-f7mvz"
Oct 30 13:02:19.776137 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.775195    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-multus-cni-dir\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:02:19.776137 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.775290    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a16e7b2-e45d-480a-a7bc-d914640b9867-tmp-dir\") pod \"node-resolver-gltng\" (UID: \"5a16e7b2-e45d-480a-a7bc-d914640b9867\") " pod="openshift-dns/node-resolver-gltng"
Oct 30 13:02:19.776137 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.775318    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-host-var-lib-cni-bin\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:02:19.776137 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.775363    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-host-run-multus-certs\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:02:19.776137 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.775372    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-multus-cni-dir\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:02:19.776137 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.775371    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-host-var-lib-cni-bin\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:02:19.776694 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.775422    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/beb2b7e4-b43d-42fc-9e8a-ecdfa6994205-konnectivity-ca\") pod \"konnectivity-agent-wpqvv\" (UID: \"beb2b7e4-b43d-42fc-9e8a-ecdfa6994205\") " pod="kube-system/konnectivity-agent-wpqvv"
Oct 30 13:02:19.776694 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.775469    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-host-run-multus-certs\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:02:19.776694 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.775555    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-ovn-node-metrics-cert\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.776694 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.776336    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/087058f5-e286-4ac7-8ed8-a280da332878-serviceca\") pod \"node-ca-f7mvz\" (UID: \"087058f5-e286-4ac7-8ed8-a280da332878\") " pod="openshift-image-registry/node-ca-f7mvz"
Oct 30 13:02:19.777041 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.776984    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1fb4abc-b161-461c-b8ef-bd1212010510-tmp\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:02:19.777041 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.777035    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f1fb4abc-b161-461c-b8ef-bd1212010510-etc-tuned\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:02:19.777272 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.777253    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/183f23e7-ae80-445b-960c-fff9b410ba5b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lrqnm\" (UID: \"183f23e7-ae80-445b-960c-fff9b410ba5b\") " pod="openshift-multus/multus-additional-cni-plugins-lrqnm"
Oct 30 13:02:19.777353 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.777324    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/de69ec51-fd3a-4bc4-ac17-ff02bc582905-multus-daemon-config\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:02:19.777412 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.777376    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/beb2b7e4-b43d-42fc-9e8a-ecdfa6994205-konnectivity-ca\") pod \"konnectivity-agent-wpqvv\" (UID: \"beb2b7e4-b43d-42fc-9e8a-ecdfa6994205\") " pod="kube-system/konnectivity-agent-wpqvv"
Oct 30 13:02:19.777467 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.777445    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/183f23e7-ae80-445b-960c-fff9b410ba5b-cni-binary-copy\") pod \"multus-additional-cni-plugins-lrqnm\" (UID: \"183f23e7-ae80-445b-960c-fff9b410ba5b\") " pod="openshift-multus/multus-additional-cni-plugins-lrqnm"
Oct 30 13:02:19.784651 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:19.784633    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 13:02:19.784720 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:19.784654    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 13:02:19.784720 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:19.784665    2576 projected.go:196] Error preparing data for projected volume kube-api-access-gz5ls for pod openshift-network-diagnostics/network-check-target-rxd99: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 13:02:19.784720 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:19.784717    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a07f5a09-920e-462f-ab6d-7a2ac53d199e-kube-api-access-gz5ls podName:a07f5a09-920e-462f-ab6d-7a2ac53d199e nodeName:}" failed. No retries permitted until 2026-10-30 13:02:20.284701136 +0000 UTC m=+2.165936358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gz5ls" (UniqueName: "kubernetes.io/projected/a07f5a09-920e-462f-ab6d-7a2ac53d199e-kube-api-access-gz5ls") pod "network-check-target-rxd99" (UID: "a07f5a09-920e-462f-ab6d-7a2ac53d199e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 13:02:19.787389 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.787362    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whwnz\" (UniqueName: \"kubernetes.io/projected/a323e312-dd91-486f-985d-92b9ab8cca04-kube-api-access-whwnz\") pod \"aws-ebs-csi-driver-node-9q945\" (UID: \"a323e312-dd91-486f-985d-92b9ab8cca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q945"
Oct 30 13:02:19.787688 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.787670    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb526\" (UniqueName: \"kubernetes.io/projected/f1fb4abc-b161-461c-b8ef-bd1212010510-kube-api-access-zb526\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:02:19.791808 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.791780    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjgw5\" (UniqueName: \"kubernetes.io/projected/183f23e7-ae80-445b-960c-fff9b410ba5b-kube-api-access-hjgw5\") pod \"multus-additional-cni-plugins-lrqnm\" (UID: \"183f23e7-ae80-445b-960c-fff9b410ba5b\") " pod="openshift-multus/multus-additional-cni-plugins-lrqnm"
Oct 30 13:02:19.792794 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.792703    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6f7b\" (UniqueName: \"kubernetes.io/projected/de69ec51-fd3a-4bc4-ac17-ff02bc582905-kube-api-access-v6f7b\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:02:19.792794 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.792726    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsfql\" (UniqueName: \"kubernetes.io/projected/a32a253a-ecaf-4f90-af1c-27a7e53b9629-kube-api-access-qsfql\") pod \"network-metrics-daemon-mbncl\" (UID: \"a32a253a-ecaf-4f90-af1c-27a7e53b9629\") " pod="openshift-multus/network-metrics-daemon-mbncl"
Oct 30 13:02:19.793624 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.793596    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gdf9\" (UniqueName: \"kubernetes.io/projected/087058f5-e286-4ac7-8ed8-a280da332878-kube-api-access-6gdf9\") pod \"node-ca-f7mvz\" (UID: \"087058f5-e286-4ac7-8ed8-a280da332878\") " pod="openshift-image-registry/node-ca-f7mvz"
Oct 30 13:02:19.794330 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.794311    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9m9z\" (UniqueName: \"kubernetes.io/projected/5a16e7b2-e45d-480a-a7bc-d914640b9867-kube-api-access-z9m9z\") pod \"node-resolver-gltng\" (UID: \"5a16e7b2-e45d-480a-a7bc-d914640b9867\") " pod="openshift-dns/node-resolver-gltng"
Oct 30 13:02:19.794449 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.794430    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgqhj\" (UniqueName: \"kubernetes.io/projected/0c8e72ed-d54d-4e6a-bf66-65a16430351d-kube-api-access-dgqhj\") pod \"iptables-alerter-5swg6\" (UID: \"0c8e72ed-d54d-4e6a-bf66-65a16430351d\") " pod="openshift-network-operator/iptables-alerter-5swg6"
Oct 30 13:02:19.794807 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.794787    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xws2p\" (UniqueName: \"kubernetes.io/projected/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-kube-api-access-xws2p\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:19.812985 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:19.812956    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87f9e67ec9b4ec6ee729521e7847a5a2.slice/crio-f2aae6460a32ba64304b33d00d3f239e0ad112f520d74613426b5ca429211604 WatchSource:0}: Error finding container f2aae6460a32ba64304b33d00d3f239e0ad112f520d74613426b5ca429211604: Status 404 returned error can't find the container with id f2aae6460a32ba64304b33d00d3f239e0ad112f520d74613426b5ca429211604
Oct 30 13:02:19.813486 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:19.813469    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5213eb0d1df39bb19f1777f20f292baf.slice/crio-9da5be1ca0326ab3d9989a5ac8b3610b3b4a0c14a8fb316621a2045016ce29a8 WatchSource:0}: Error finding container 9da5be1ca0326ab3d9989a5ac8b3610b3b4a0c14a8fb316621a2045016ce29a8: Status 404 returned error can't find the container with id 9da5be1ca0326ab3d9989a5ac8b3610b3b4a0c14a8fb316621a2045016ce29a8
Oct 30 13:02:19.818261 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.818242    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 13:02:19.905090 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.905048    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-f7mvz"
Oct 30 13:02:19.915967 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:19.915940    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod087058f5_e286_4ac7_8ed8_a280da332878.slice/crio-2e963d5ab87bb8f8412d3ed93ae6eef220d880e9b5a0350f3f0c8365b9b5d6b2 WatchSource:0}: Error finding container 2e963d5ab87bb8f8412d3ed93ae6eef220d880e9b5a0350f3f0c8365b9b5d6b2: Status 404 returned error can't find the container with id 2e963d5ab87bb8f8412d3ed93ae6eef220d880e9b5a0350f3f0c8365b9b5d6b2
Oct 30 13:02:19.928990 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.928966    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lrqnm"
Oct 30 13:02:19.930173 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.930152    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-f2bqp"
Oct 30 13:02:19.940554 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.940529    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wpqvv"
Oct 30 13:02:19.941890 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:19.941863    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod183f23e7_ae80_445b_960c_fff9b410ba5b.slice/crio-c6bb98ae54b0111cebebb0ab298186b328e44754b62c8032028fd66414186c50 WatchSource:0}: Error finding container c6bb98ae54b0111cebebb0ab298186b328e44754b62c8032028fd66414186c50: Status 404 returned error can't find the container with id c6bb98ae54b0111cebebb0ab298186b328e44754b62c8032028fd66414186c50
Oct 30 13:02:19.942676 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:19.942652    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde69ec51_fd3a_4bc4_ac17_ff02bc582905.slice/crio-a911208b7a7070410d4ca2f43e084cbbbc08ab928ec7df2bb1a3524f92dc8607 WatchSource:0}: Error finding container a911208b7a7070410d4ca2f43e084cbbbc08ab928ec7df2bb1a3524f92dc8607: Status 404 returned error can't find the container with id a911208b7a7070410d4ca2f43e084cbbbc08ab928ec7df2bb1a3524f92dc8607
Oct 30 13:02:19.950915 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:19.950886    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeb2b7e4_b43d_42fc_9e8a_ecdfa6994205.slice/crio-2f036d26baa035a5f2f97f9c748b0997a23f90fc3a79944f898e4fc7ebf17f73 WatchSource:0}: Error finding container 2f036d26baa035a5f2f97f9c748b0997a23f90fc3a79944f898e4fc7ebf17f73: Status 404 returned error can't find the container with id 2f036d26baa035a5f2f97f9c748b0997a23f90fc3a79944f898e4fc7ebf17f73
Oct 30 13:02:19.955316 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.955294    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gltng"
Oct 30 13:02:19.964858 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:19.964836    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a16e7b2_e45d_480a_a7bc_d914640b9867.slice/crio-4e3de8cc64e483689b44b655b9983ba4b90579eb7514936c00f38d8a8d37bed6 WatchSource:0}: Error finding container 4e3de8cc64e483689b44b655b9983ba4b90579eb7514936c00f38d8a8d37bed6: Status 404 returned error can't find the container with id 4e3de8cc64e483689b44b655b9983ba4b90579eb7514936c00f38d8a8d37bed6
Oct 30 13:02:19.997055 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.997010    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5swg6"
Oct 30 13:02:19.999362 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:19.999338    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q945"
Oct 30 13:02:20.000679 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:20.000656    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:20.006916 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:20.006899    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:02:20.007562 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:20.007538    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c8e72ed_d54d_4e6a_bf66_65a16430351d.slice/crio-a9872fe8d3163787d5f6fcf7d35449ec1edca5abd3d826b8108c491c819a3783 WatchSource:0}: Error finding container a9872fe8d3163787d5f6fcf7d35449ec1edca5abd3d826b8108c491c819a3783: Status 404 returned error can't find the container with id a9872fe8d3163787d5f6fcf7d35449ec1edca5abd3d826b8108c491c819a3783
Oct 30 13:02:20.010954 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:20.010924    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a675ac7_fbe1_449b_accf_bb0456ed0bd3.slice/crio-0c41bc03f6369e26cb0fac4e724d818f33ed871dba80d4d83b87b5bd08b7c28f WatchSource:0}: Error finding container 0c41bc03f6369e26cb0fac4e724d818f33ed871dba80d4d83b87b5bd08b7c28f: Status 404 returned error can't find the container with id 0c41bc03f6369e26cb0fac4e724d818f33ed871dba80d4d83b87b5bd08b7c28f
Oct 30 13:02:20.014667 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:20.014639    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda323e312_dd91_486f_985d_92b9ab8cca04.slice/crio-f1de5a011f5645942aa0dd822f5aae0ca5197e9b5b01aacafd9fa403fe7ed36d WatchSource:0}: Error finding container f1de5a011f5645942aa0dd822f5aae0ca5197e9b5b01aacafd9fa403fe7ed36d: Status 404 returned error can't find the container with id f1de5a011f5645942aa0dd822f5aae0ca5197e9b5b01aacafd9fa403fe7ed36d
Oct 30 13:02:20.017627 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:20.017604    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1fb4abc_b161_461c_b8ef_bd1212010510.slice/crio-a03d97bc9dd047e45f1d5c8826b21dd5d16660967c022f284e71aa767c0bd992 WatchSource:0}: Error finding container a03d97bc9dd047e45f1d5c8826b21dd5d16660967c022f284e71aa767c0bd992: Status 404 returned error can't find the container with id a03d97bc9dd047e45f1d5c8826b21dd5d16660967c022f284e71aa767c0bd992
Oct 30 13:02:20.280078 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:20.279984    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a32a253a-ecaf-4f90-af1c-27a7e53b9629-metrics-certs\") pod \"network-metrics-daemon-mbncl\" (UID: \"a32a253a-ecaf-4f90-af1c-27a7e53b9629\") " pod="openshift-multus/network-metrics-daemon-mbncl"
Oct 30 13:02:20.280078 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:20.280046    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/10b774a1-1476-4f91-b6e5-d52b3f45700e-original-pull-secret\") pod \"global-pull-secret-syncer-wrfqz\" (UID: \"10b774a1-1476-4f91-b6e5-d52b3f45700e\") " pod="kube-system/global-pull-secret-syncer-wrfqz"
Oct 30 13:02:20.280329 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:20.280197    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 13:02:20.280329 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:20.280267    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10b774a1-1476-4f91-b6e5-d52b3f45700e-original-pull-secret podName:10b774a1-1476-4f91-b6e5-d52b3f45700e nodeName:}" failed. No retries permitted until 2026-10-30 13:02:21.280247411 +0000 UTC m=+3.161482620 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/10b774a1-1476-4f91-b6e5-d52b3f45700e-original-pull-secret") pod "global-pull-secret-syncer-wrfqz" (UID: "10b774a1-1476-4f91-b6e5-d52b3f45700e") : object "kube-system"/"original-pull-secret" not registered
Oct 30 13:02:20.280452 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:20.280341    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 13:02:20.280452 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:20.280375    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a32a253a-ecaf-4f90-af1c-27a7e53b9629-metrics-certs podName:a32a253a-ecaf-4f90-af1c-27a7e53b9629 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:21.280363521 +0000 UTC m=+3.161598722 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a32a253a-ecaf-4f90-af1c-27a7e53b9629-metrics-certs") pod "network-metrics-daemon-mbncl" (UID: "a32a253a-ecaf-4f90-af1c-27a7e53b9629") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 13:02:20.381422 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:20.381381    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz5ls\" (UniqueName: \"kubernetes.io/projected/a07f5a09-920e-462f-ab6d-7a2ac53d199e-kube-api-access-gz5ls\") pod \"network-check-target-rxd99\" (UID: \"a07f5a09-920e-462f-ab6d-7a2ac53d199e\") " pod="openshift-network-diagnostics/network-check-target-rxd99"
Oct 30 13:02:20.381657 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:20.381632    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 13:02:20.381739 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:20.381662    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 13:02:20.381739 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:20.381676    2576 projected.go:196] Error preparing data for projected volume kube-api-access-gz5ls for pod openshift-network-diagnostics/network-check-target-rxd99: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 13:02:20.381844 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:20.381740    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a07f5a09-920e-462f-ab6d-7a2ac53d199e-kube-api-access-gz5ls podName:a07f5a09-920e-462f-ab6d-7a2ac53d199e nodeName:}" failed. No retries permitted until 2026-10-30 13:02:21.381722316 +0000 UTC m=+3.262957531 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gz5ls" (UniqueName: "kubernetes.io/projected/a07f5a09-920e-462f-ab6d-7a2ac53d199e-kube-api-access-gz5ls") pod "network-check-target-rxd99" (UID: "a07f5a09-920e-462f-ab6d-7a2ac53d199e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 13:02:20.421849 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:20.421810    2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 13:02:20.454089 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:20.454055    2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 13:02:20.611015 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:20.610889    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 12:57:20 +0000 UTC" deadline="2028-06-12 09:01:40.004316767 +0000 UTC"
Oct 30 13:02:20.611015 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:20.610959    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14179h59m19.39336223s"
Oct 30 13:02:20.684059 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:20.683991    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s9vk9" event={"ID":"f1fb4abc-b161-461c-b8ef-bd1212010510","Type":"ContainerStarted","Data":"a03d97bc9dd047e45f1d5c8826b21dd5d16660967c022f284e71aa767c0bd992"}
Oct 30 13:02:20.697529 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:20.697484    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5swg6" event={"ID":"0c8e72ed-d54d-4e6a-bf66-65a16430351d","Type":"ContainerStarted","Data":"a9872fe8d3163787d5f6fcf7d35449ec1edca5abd3d826b8108c491c819a3783"}
Oct 30 13:02:20.705659 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:20.705612    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wpqvv" event={"ID":"beb2b7e4-b43d-42fc-9e8a-ecdfa6994205","Type":"ContainerStarted","Data":"2f036d26baa035a5f2f97f9c748b0997a23f90fc3a79944f898e4fc7ebf17f73"}
Oct 30 13:02:20.711806 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:20.711766    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lrqnm" event={"ID":"183f23e7-ae80-445b-960c-fff9b410ba5b","Type":"ContainerStarted","Data":"c6bb98ae54b0111cebebb0ab298186b328e44754b62c8032028fd66414186c50"}
Oct 30 13:02:20.729208 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:20.729156    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-f7mvz" event={"ID":"087058f5-e286-4ac7-8ed8-a280da332878","Type":"ContainerStarted","Data":"2e963d5ab87bb8f8412d3ed93ae6eef220d880e9b5a0350f3f0c8365b9b5d6b2"}
Oct 30 13:02:20.738234 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:20.738180    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-31.ec2.internal" event={"ID":"5213eb0d1df39bb19f1777f20f292baf","Type":"ContainerStarted","Data":"9da5be1ca0326ab3d9989a5ac8b3610b3b4a0c14a8fb316621a2045016ce29a8"}
Oct 30 13:02:20.751400 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:20.751309    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q945" event={"ID":"a323e312-dd91-486f-985d-92b9ab8cca04","Type":"ContainerStarted","Data":"f1de5a011f5645942aa0dd822f5aae0ca5197e9b5b01aacafd9fa403fe7ed36d"}
Oct 30 13:02:20.753625 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:20.753590    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-67tnm" event={"ID":"3a675ac7-fbe1-449b-accf-bb0456ed0bd3","Type":"ContainerStarted","Data":"0c41bc03f6369e26cb0fac4e724d818f33ed871dba80d4d83b87b5bd08b7c28f"}
Oct 30 13:02:20.761491 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:20.761458    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gltng" event={"ID":"5a16e7b2-e45d-480a-a7bc-d914640b9867","Type":"ContainerStarted","Data":"4e3de8cc64e483689b44b655b9983ba4b90579eb7514936c00f38d8a8d37bed6"}
Oct 30 13:02:20.766186 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:20.766153    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-f2bqp" event={"ID":"de69ec51-fd3a-4bc4-ac17-ff02bc582905","Type":"ContainerStarted","Data":"a911208b7a7070410d4ca2f43e084cbbbc08ab928ec7df2bb1a3524f92dc8607"}
Oct 30 13:02:20.772393 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:20.772362    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-31.ec2.internal" event={"ID":"87f9e67ec9b4ec6ee729521e7847a5a2","Type":"ContainerStarted","Data":"f2aae6460a32ba64304b33d00d3f239e0ad112f520d74613426b5ca429211604"}
Oct 30 13:02:20.899903 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:20.899810    2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 13:02:21.289996 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:21.289928    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/10b774a1-1476-4f91-b6e5-d52b3f45700e-original-pull-secret\") pod \"global-pull-secret-syncer-wrfqz\" (UID: \"10b774a1-1476-4f91-b6e5-d52b3f45700e\") " pod="kube-system/global-pull-secret-syncer-wrfqz"
Oct 30 13:02:21.290220 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:21.290049    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a32a253a-ecaf-4f90-af1c-27a7e53b9629-metrics-certs\") pod \"network-metrics-daemon-mbncl\" (UID: \"a32a253a-ecaf-4f90-af1c-27a7e53b9629\") " pod="openshift-multus/network-metrics-daemon-mbncl"
Oct 30 13:02:21.290220 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:21.290208    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 13:02:21.290407 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:21.290285    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a32a253a-ecaf-4f90-af1c-27a7e53b9629-metrics-certs podName:a32a253a-ecaf-4f90-af1c-27a7e53b9629 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:23.29026418 +0000 UTC m=+5.171499400 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a32a253a-ecaf-4f90-af1c-27a7e53b9629-metrics-certs") pod "network-metrics-daemon-mbncl" (UID: "a32a253a-ecaf-4f90-af1c-27a7e53b9629") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 13:02:21.290893 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:21.290721    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 13:02:21.290893 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:21.290781    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10b774a1-1476-4f91-b6e5-d52b3f45700e-original-pull-secret podName:10b774a1-1476-4f91-b6e5-d52b3f45700e nodeName:}" failed. No retries permitted until 2026-10-30 13:02:23.290764273 +0000 UTC m=+5.171999477 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/10b774a1-1476-4f91-b6e5-d52b3f45700e-original-pull-secret") pod "global-pull-secret-syncer-wrfqz" (UID: "10b774a1-1476-4f91-b6e5-d52b3f45700e") : object "kube-system"/"original-pull-secret" not registered
Oct 30 13:02:21.391648 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:21.390929    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz5ls\" (UniqueName: \"kubernetes.io/projected/a07f5a09-920e-462f-ab6d-7a2ac53d199e-kube-api-access-gz5ls\") pod \"network-check-target-rxd99\" (UID: \"a07f5a09-920e-462f-ab6d-7a2ac53d199e\") " pod="openshift-network-diagnostics/network-check-target-rxd99"
Oct 30 13:02:21.391648 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:21.391152    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 13:02:21.391648 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:21.391189    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 13:02:21.391648 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:21.391205    2576 projected.go:196] Error preparing data for projected volume kube-api-access-gz5ls for pod openshift-network-diagnostics/network-check-target-rxd99: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 13:02:21.391648 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:21.391273    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a07f5a09-920e-462f-ab6d-7a2ac53d199e-kube-api-access-gz5ls podName:a07f5a09-920e-462f-ab6d-7a2ac53d199e nodeName:}" failed. No retries permitted until 2026-10-30 13:02:23.391255076 +0000 UTC m=+5.272490275 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gz5ls" (UniqueName: "kubernetes.io/projected/a07f5a09-920e-462f-ab6d-7a2ac53d199e-kube-api-access-gz5ls") pod "network-check-target-rxd99" (UID: "a07f5a09-920e-462f-ab6d-7a2ac53d199e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 13:02:21.611914 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:21.611811    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 12:57:20 +0000 UTC" deadline="2028-06-28 22:23:26.056551575 +0000 UTC"
Oct 30 13:02:21.611914 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:21.611862    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14577h21m4.444696301s"
Oct 30 13:02:21.633968 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:21.633931    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrfqz"
Oct 30 13:02:21.634170 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:21.634085    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrfqz" podUID="10b774a1-1476-4f91-b6e5-d52b3f45700e"
Oct 30 13:02:21.634582 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:21.634559    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxd99"
Oct 30 13:02:21.634693 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:21.634669    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rxd99" podUID="a07f5a09-920e-462f-ab6d-7a2ac53d199e"
Oct 30 13:02:21.634764 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:21.634749    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbncl"
Oct 30 13:02:21.634872 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:21.634849    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mbncl" podUID="a32a253a-ecaf-4f90-af1c-27a7e53b9629"
Oct 30 13:02:23.309267 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:23.308368    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a32a253a-ecaf-4f90-af1c-27a7e53b9629-metrics-certs\") pod \"network-metrics-daemon-mbncl\" (UID: \"a32a253a-ecaf-4f90-af1c-27a7e53b9629\") " pod="openshift-multus/network-metrics-daemon-mbncl"
Oct 30 13:02:23.309267 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:23.308432    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/10b774a1-1476-4f91-b6e5-d52b3f45700e-original-pull-secret\") pod \"global-pull-secret-syncer-wrfqz\" (UID: \"10b774a1-1476-4f91-b6e5-d52b3f45700e\") " pod="kube-system/global-pull-secret-syncer-wrfqz"
Oct 30 13:02:23.309267 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:23.308639    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 13:02:23.309267 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:23.308709    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10b774a1-1476-4f91-b6e5-d52b3f45700e-original-pull-secret podName:10b774a1-1476-4f91-b6e5-d52b3f45700e nodeName:}" failed. No retries permitted until 2026-10-30 13:02:27.3086889 +0000 UTC m=+9.189924119 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/10b774a1-1476-4f91-b6e5-d52b3f45700e-original-pull-secret") pod "global-pull-secret-syncer-wrfqz" (UID: "10b774a1-1476-4f91-b6e5-d52b3f45700e") : object "kube-system"/"original-pull-secret" not registered
Oct 30 13:02:23.309267 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:23.309151    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 13:02:23.310225 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:23.310179    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a32a253a-ecaf-4f90-af1c-27a7e53b9629-metrics-certs podName:a32a253a-ecaf-4f90-af1c-27a7e53b9629 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:27.309191149 +0000 UTC m=+9.190426365 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a32a253a-ecaf-4f90-af1c-27a7e53b9629-metrics-certs") pod "network-metrics-daemon-mbncl" (UID: "a32a253a-ecaf-4f90-af1c-27a7e53b9629") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 13:02:23.409962 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:23.409886    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz5ls\" (UniqueName: \"kubernetes.io/projected/a07f5a09-920e-462f-ab6d-7a2ac53d199e-kube-api-access-gz5ls\") pod \"network-check-target-rxd99\" (UID: \"a07f5a09-920e-462f-ab6d-7a2ac53d199e\") " pod="openshift-network-diagnostics/network-check-target-rxd99"
Oct 30 13:02:23.410142 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:23.410121    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 13:02:23.410203 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:23.410149    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 13:02:23.410203 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:23.410164    2576 projected.go:196] Error preparing data for projected volume kube-api-access-gz5ls for pod openshift-network-diagnostics/network-check-target-rxd99: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 13:02:23.410362 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:23.410286    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a07f5a09-920e-462f-ab6d-7a2ac53d199e-kube-api-access-gz5ls podName:a07f5a09-920e-462f-ab6d-7a2ac53d199e nodeName:}" failed. No retries permitted until 2026-10-30 13:02:27.410265181 +0000 UTC m=+9.291500398 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gz5ls" (UniqueName: "kubernetes.io/projected/a07f5a09-920e-462f-ab6d-7a2ac53d199e-kube-api-access-gz5ls") pod "network-check-target-rxd99" (UID: "a07f5a09-920e-462f-ab6d-7a2ac53d199e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 13:02:23.634594 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:23.634077    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxd99"
Oct 30 13:02:23.634594 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:23.634206    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rxd99" podUID="a07f5a09-920e-462f-ab6d-7a2ac53d199e"
Oct 30 13:02:23.634594 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:23.634302    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrfqz"
Oct 30 13:02:23.634594 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:23.634438    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrfqz" podUID="10b774a1-1476-4f91-b6e5-d52b3f45700e"
Oct 30 13:02:23.634594 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:23.634496    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbncl"
Oct 30 13:02:23.634594 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:23.634576    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mbncl" podUID="a32a253a-ecaf-4f90-af1c-27a7e53b9629"
Oct 30 13:02:24.813377 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:24.813078    2576 generic.go:356] "Generic (PLEG): container finished" podID="183f23e7-ae80-445b-960c-fff9b410ba5b" containerID="3dbef6f614e977c11d58740c2274d9ac1c89bd6ab385fd78cc465502e249bc46" exitCode=0
Oct 30 13:02:24.813798 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:24.813442    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lrqnm" event={"ID":"183f23e7-ae80-445b-960c-fff9b410ba5b","Type":"ContainerDied","Data":"3dbef6f614e977c11d58740c2274d9ac1c89bd6ab385fd78cc465502e249bc46"}
Oct 30 13:02:24.828113 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:24.828078    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-31.ec2.internal" event={"ID":"5213eb0d1df39bb19f1777f20f292baf","Type":"ContainerStarted","Data":"0018ddbe4e6d2cdcf0b570d1855a68a1f076c0856c42d4940fa6b1e725e6c986"}
Oct 30 13:02:24.830936 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:24.830895    2576 generic.go:356] "Generic (PLEG): container finished" podID="87f9e67ec9b4ec6ee729521e7847a5a2" containerID="373f8b51d46bf6fe5f41a142dc388ae8a641cd6b7d0bd42bfcc161e8d6608e65" exitCode=0
Oct 30 13:02:24.830936 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:24.830933    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-31.ec2.internal" event={"ID":"87f9e67ec9b4ec6ee729521e7847a5a2","Type":"ContainerDied","Data":"373f8b51d46bf6fe5f41a142dc388ae8a641cd6b7d0bd42bfcc161e8d6608e65"}
Oct 30 13:02:25.633762 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:25.633727    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrfqz"
Oct 30 13:02:25.633943 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:25.633869    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrfqz" podUID="10b774a1-1476-4f91-b6e5-d52b3f45700e"
Oct 30 13:02:25.634378 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:25.634314    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbncl"
Oct 30 13:02:25.634591 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:25.634506    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mbncl" podUID="a32a253a-ecaf-4f90-af1c-27a7e53b9629"
Oct 30 13:02:25.634591 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:25.634547    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxd99"
Oct 30 13:02:25.634778 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:25.634663    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rxd99" podUID="a07f5a09-920e-462f-ab6d-7a2ac53d199e"
Oct 30 13:02:25.838679 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:25.838065    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-31.ec2.internal" event={"ID":"87f9e67ec9b4ec6ee729521e7847a5a2","Type":"ContainerStarted","Data":"6e39d68b4ab004a78973391e1a1585e868d71d91f7dfd95d6c2aa4562d035695"}
Oct 30 13:02:25.844528 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:25.843793    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wpqvv" event={"ID":"beb2b7e4-b43d-42fc-9e8a-ecdfa6994205","Type":"ContainerStarted","Data":"eb7de3d6a39ed62cdde2f9f4376b451b6634064f7bb8f9940c8faa243a4edc36"}
Oct 30 13:02:25.850427 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:25.850249    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-f7mvz" event={"ID":"087058f5-e286-4ac7-8ed8-a280da332878","Type":"ContainerStarted","Data":"3fd5d99dd61aea1b752b48a1723ed78ca9d62afd36a278d5ad5e536c3a464375"}
Oct 30 13:02:25.876303 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:25.876232    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-31.ec2.internal" podStartSLOduration=6.876192122 podStartE2EDuration="6.876192122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:02:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:24.951409286 +0000 UTC m=+6.832644511" watchObservedRunningTime="2026-10-30 13:02:25.876192122 +0000 UTC m=+7.757427347"
Oct 30 13:02:25.915519 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:25.915452    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-31.ec2.internal" podStartSLOduration=6.915425569 podStartE2EDuration="6.915425569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:02:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:25.876933586 +0000 UTC m=+7.758168810" watchObservedRunningTime="2026-10-30 13:02:25.915425569 +0000 UTC m=+7.796660794"
Oct 30 13:02:25.954327 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:25.954258    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wpqvv" podStartSLOduration=2.32030824 podStartE2EDuration="6.954233263s" totalImagesPullingTime="4.633925023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:25.915889915 +0000 UTC m=+7.797125139" watchObservedRunningTime="2026-10-30 13:02:25.954233263 +0000 UTC m=+7.835468490"
Oct 30 13:02:26.825772 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:26.825689    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wpqvv"
Oct 30 13:02:26.826701 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:26.826482    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wpqvv"
Oct 30 13:02:26.869128 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:26.867722    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-f7mvz" podStartSLOduration=2.191697077 podStartE2EDuration="7.867702347s" totalImagesPullingTime="5.67600527s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:02:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:25.959561384 +0000 UTC m=+7.840796622" watchObservedRunningTime="2026-10-30 13:02:26.867702347 +0000 UTC m=+8.748937572"
Oct 30 13:02:27.344873 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:27.344824    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a32a253a-ecaf-4f90-af1c-27a7e53b9629-metrics-certs\") pod \"network-metrics-daemon-mbncl\" (UID: \"a32a253a-ecaf-4f90-af1c-27a7e53b9629\") " pod="openshift-multus/network-metrics-daemon-mbncl"
Oct 30 13:02:27.345085 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:27.344883    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/10b774a1-1476-4f91-b6e5-d52b3f45700e-original-pull-secret\") pod \"global-pull-secret-syncer-wrfqz\" (UID: \"10b774a1-1476-4f91-b6e5-d52b3f45700e\") " pod="kube-system/global-pull-secret-syncer-wrfqz"
Oct 30 13:02:27.345225 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:27.345116    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 13:02:27.345225 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:27.345189    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10b774a1-1476-4f91-b6e5-d52b3f45700e-original-pull-secret podName:10b774a1-1476-4f91-b6e5-d52b3f45700e nodeName:}" failed. No retries permitted until 2026-10-30 13:02:35.34516879 +0000 UTC m=+17.226404008 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/10b774a1-1476-4f91-b6e5-d52b3f45700e-original-pull-secret") pod "global-pull-secret-syncer-wrfqz" (UID: "10b774a1-1476-4f91-b6e5-d52b3f45700e") : object "kube-system"/"original-pull-secret" not registered
Oct 30 13:02:27.345361 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:27.345283    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 13:02:27.345427 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:27.345339    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a32a253a-ecaf-4f90-af1c-27a7e53b9629-metrics-certs podName:a32a253a-ecaf-4f90-af1c-27a7e53b9629 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:35.345326738 +0000 UTC m=+17.226561954 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a32a253a-ecaf-4f90-af1c-27a7e53b9629-metrics-certs") pod "network-metrics-daemon-mbncl" (UID: "a32a253a-ecaf-4f90-af1c-27a7e53b9629") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 13:02:27.445407 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:27.445359    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz5ls\" (UniqueName: \"kubernetes.io/projected/a07f5a09-920e-462f-ab6d-7a2ac53d199e-kube-api-access-gz5ls\") pod \"network-check-target-rxd99\" (UID: \"a07f5a09-920e-462f-ab6d-7a2ac53d199e\") " pod="openshift-network-diagnostics/network-check-target-rxd99"
Oct 30 13:02:27.445628 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:27.445608    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 13:02:27.445694 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:27.445636    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 13:02:27.445694 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:27.445652    2576 projected.go:196] Error preparing data for projected volume kube-api-access-gz5ls for pod openshift-network-diagnostics/network-check-target-rxd99: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 13:02:27.445807 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:27.445719    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a07f5a09-920e-462f-ab6d-7a2ac53d199e-kube-api-access-gz5ls podName:a07f5a09-920e-462f-ab6d-7a2ac53d199e nodeName:}" failed. No retries permitted until 2026-10-30 13:02:35.445698422 +0000 UTC m=+17.326933647 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gz5ls" (UniqueName: "kubernetes.io/projected/a07f5a09-920e-462f-ab6d-7a2ac53d199e-kube-api-access-gz5ls") pod "network-check-target-rxd99" (UID: "a07f5a09-920e-462f-ab6d-7a2ac53d199e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 13:02:27.633944 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:27.633901    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxd99"
Oct 30 13:02:27.634112 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:27.634044    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rxd99" podUID="a07f5a09-920e-462f-ab6d-7a2ac53d199e"
Oct 30 13:02:27.634530 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:27.634505    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbncl"
Oct 30 13:02:27.634655 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:27.634630    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mbncl" podUID="a32a253a-ecaf-4f90-af1c-27a7e53b9629"
Oct 30 13:02:27.634719 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:27.634703    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrfqz"
Oct 30 13:02:27.634833 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:27.634780    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrfqz" podUID="10b774a1-1476-4f91-b6e5-d52b3f45700e"
Oct 30 13:02:27.861649 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:27.861555    2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 13:02:27.862156 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:27.862123    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q945" event={"ID":"a323e312-dd91-486f-985d-92b9ab8cca04","Type":"ContainerStarted","Data":"743115c954a79565ef082db35fd01a429591cdaa91dac8665c15dfc2a08b4b55"}
Oct 30 13:02:28.434949 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:28.434910    2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 13:02:28.637244 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:28.631558    2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T13:02:28.434924824Z","UUID":"4e826e05-851a-4acc-8b67-3a3126e97960","Handler":null,"Name":"","Endpoint":""}
Oct 30 13:02:28.637244 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:28.635930    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 13:02:28.637244 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:28.635958    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 13:02:28.872209 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:28.872175    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q945" event={"ID":"a323e312-dd91-486f-985d-92b9ab8cca04","Type":"ContainerStarted","Data":"99d3fc003b68b4155cef42d6aaabdf34eaeb19a75ea7da8c35ff2002d803cbed"}
Oct 30 13:02:28.874907 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:28.874849    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-f2bqp" event={"ID":"de69ec51-fd3a-4bc4-ac17-ff02bc582905","Type":"ContainerStarted","Data":"0baa9ab138bce8ccfa02e65f4a0c6c360a00e32250c9d6a3726acdf0673450b7"}
Oct 30 13:02:28.916825 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:28.916752    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-f2bqp" podStartSLOduration=1.345267467 podStartE2EDuration="9.916733919s" totalImagesPullingTime="8.571466452s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:02:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:28.915802809 +0000 UTC m=+10.797038033" watchObservedRunningTime="2026-10-30 13:02:28.916733919 +0000 UTC m=+10.797969140"
Oct 30 13:02:29.634041 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:29.633959    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrfqz"
Oct 30 13:02:29.634041 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:29.633986    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbncl"
Oct 30 13:02:29.634041 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:29.633988    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxd99"
Oct 30 13:02:29.634543 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:29.634107    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrfqz" podUID="10b774a1-1476-4f91-b6e5-d52b3f45700e"
Oct 30 13:02:29.634543 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:29.634199    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rxd99" podUID="a07f5a09-920e-462f-ab6d-7a2ac53d199e"
Oct 30 13:02:29.634543 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:29.634318    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mbncl" podUID="a32a253a-ecaf-4f90-af1c-27a7e53b9629"
Oct 30 13:02:29.879585 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:29.879545    2576 generic.go:356] "Generic (PLEG): container finished" podID="183f23e7-ae80-445b-960c-fff9b410ba5b" containerID="62fbb027596ba7b31ae1480cfd43f7c21605927721260e28537ce8164858fc50" exitCode=0
Oct 30 13:02:29.879764 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:29.879649    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lrqnm" event={"ID":"183f23e7-ae80-445b-960c-fff9b410ba5b","Type":"ContainerDied","Data":"62fbb027596ba7b31ae1480cfd43f7c21605927721260e28537ce8164858fc50"}
Oct 30 13:02:29.883222 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:29.883157    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q945" event={"ID":"a323e312-dd91-486f-985d-92b9ab8cca04","Type":"ContainerStarted","Data":"e48646452b2a615bc6948490621b41c2116dd4cccdea33bb7657d4210c732113"}
Oct 30 13:02:29.884731 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:29.884657    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gltng" event={"ID":"5a16e7b2-e45d-480a-a7bc-d914640b9867","Type":"ContainerStarted","Data":"7232be289649f32a8ee23475cf5312b22951a97c054dd1ddf3ed795bed3cc95d"}
Oct 30 13:02:29.996019 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:29.995385    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-gltng" podStartSLOduration=2.132695177 podStartE2EDuration="10.995363972s" totalImagesPullingTime="8.862668795s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:02:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:29.957334704 +0000 UTC m=+11.838569943" watchObservedRunningTime="2026-10-30 13:02:29.995363972 +0000 UTC m=+11.876599194"
Oct 30 13:02:30.890405 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:30.890361    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5swg6" event={"ID":"0c8e72ed-d54d-4e6a-bf66-65a16430351d","Type":"ContainerStarted","Data":"0bd7c675966a3953ce6e18561208afe64132f3a4d5eded2afe8e82c2ca27a7a1"}
Oct 30 13:02:30.893170 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:30.893123    2576 generic.go:356] "Generic (PLEG): container finished" podID="183f23e7-ae80-445b-960c-fff9b410ba5b" containerID="2a8648ce69c555e4848b8163c06040a075d2defd78298a829108a0e7d2efb315" exitCode=0
Oct 30 13:02:30.893310 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:30.893162    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lrqnm" event={"ID":"183f23e7-ae80-445b-960c-fff9b410ba5b","Type":"ContainerDied","Data":"2a8648ce69c555e4848b8163c06040a075d2defd78298a829108a0e7d2efb315"}
Oct 30 13:02:30.930685 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:30.930625    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q945" podStartSLOduration=2.86979294 podStartE2EDuration="11.930604835s" totalImagesPullingTime="9.060811895s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:02:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:29.997476776 +0000 UTC m=+11.878711996" watchObservedRunningTime="2026-10-30 13:02:30.930604835 +0000 UTC m=+12.811840057"
Oct 30 13:02:30.930857 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:30.930827    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5swg6" podStartSLOduration=2.59771254 podStartE2EDuration="11.930819958s" totalImagesPullingTime="9.333107418s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:02:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:30.929663835 +0000 UTC m=+12.810899056" watchObservedRunningTime="2026-10-30 13:02:30.930819958 +0000 UTC m=+12.812055182"
Oct 30 13:02:31.633326 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:31.633292    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrfqz"
Oct 30 13:02:31.633488 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:31.633308    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxd99"
Oct 30 13:02:31.633488 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:31.633457    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrfqz" podUID="10b774a1-1476-4f91-b6e5-d52b3f45700e"
Oct 30 13:02:31.633603 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:31.633504    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbncl"
Oct 30 13:02:31.633603 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:31.633584    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rxd99" podUID="a07f5a09-920e-462f-ab6d-7a2ac53d199e"
Oct 30 13:02:31.634037 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:31.634010    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mbncl" podUID="a32a253a-ecaf-4f90-af1c-27a7e53b9629"
Oct 30 13:02:31.898922 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:31.898846    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s9vk9" event={"ID":"f1fb4abc-b161-461c-b8ef-bd1212010510","Type":"ContainerStarted","Data":"e58c8ec0f6c955419d335cec247a2ab2ff56aeb3e627b60780fbd2aa9316a745"}
Oct 30 13:02:31.938070 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:31.938023    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-s9vk9" podStartSLOduration=2.027216221 podStartE2EDuration="12.938006793s" totalImagesPullingTime="10.910790572s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:02:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:31.937775458 +0000 UTC m=+13.819010679" watchObservedRunningTime="2026-10-30 13:02:31.938006793 +0000 UTC m=+13.819242012"
Oct 30 13:02:32.904200 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:32.904055    2576 generic.go:356] "Generic (PLEG): container finished" podID="183f23e7-ae80-445b-960c-fff9b410ba5b" containerID="c758bf56aa652a62088df4a941716f0043730bd5e00586c7eba31a97f25e8092" exitCode=0
Oct 30 13:02:32.904200 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:32.904143    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lrqnm" event={"ID":"183f23e7-ae80-445b-960c-fff9b410ba5b","Type":"ContainerDied","Data":"c758bf56aa652a62088df4a941716f0043730bd5e00586c7eba31a97f25e8092"}
Oct 30 13:02:33.633850 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:33.633815    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrfqz"
Oct 30 13:02:33.634035 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:33.633957    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrfqz" podUID="10b774a1-1476-4f91-b6e5-d52b3f45700e"
Oct 30 13:02:33.634298 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:33.634273    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxd99"
Oct 30 13:02:33.634411 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:33.634361    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rxd99" podUID="a07f5a09-920e-462f-ab6d-7a2ac53d199e"
Oct 30 13:02:33.634469 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:33.634416    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbncl"
Oct 30 13:02:33.634517 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:33.634494    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mbncl" podUID="a32a253a-ecaf-4f90-af1c-27a7e53b9629"
Oct 30 13:02:34.912604 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:34.912300    2576 generic.go:356] "Generic (PLEG): container finished" podID="183f23e7-ae80-445b-960c-fff9b410ba5b" containerID="b94baaa2927291a4b20a262129ceaf5309bbfe071d8a90f62f2e6b53d1c81c81" exitCode=0
Oct 30 13:02:34.912604 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:34.912383    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lrqnm" event={"ID":"183f23e7-ae80-445b-960c-fff9b410ba5b","Type":"ContainerDied","Data":"b94baaa2927291a4b20a262129ceaf5309bbfe071d8a90f62f2e6b53d1c81c81"}
Oct 30 13:02:35.408022 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:35.408000    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a32a253a-ecaf-4f90-af1c-27a7e53b9629-metrics-certs\") pod \"network-metrics-daemon-mbncl\" (UID: \"a32a253a-ecaf-4f90-af1c-27a7e53b9629\") " pod="openshift-multus/network-metrics-daemon-mbncl"
Oct 30 13:02:35.408169 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:35.408037    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/10b774a1-1476-4f91-b6e5-d52b3f45700e-original-pull-secret\") pod \"global-pull-secret-syncer-wrfqz\" (UID: \"10b774a1-1476-4f91-b6e5-d52b3f45700e\") " pod="kube-system/global-pull-secret-syncer-wrfqz"
Oct 30 13:02:35.408169 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:35.408158    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 13:02:35.408233 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:35.408199    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10b774a1-1476-4f91-b6e5-d52b3f45700e-original-pull-secret podName:10b774a1-1476-4f91-b6e5-d52b3f45700e nodeName:}" failed. No retries permitted until 2026-10-30 13:02:51.408187366 +0000 UTC m=+33.289422564 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/10b774a1-1476-4f91-b6e5-d52b3f45700e-original-pull-secret") pod "global-pull-secret-syncer-wrfqz" (UID: "10b774a1-1476-4f91-b6e5-d52b3f45700e") : object "kube-system"/"original-pull-secret" not registered
Oct 30 13:02:35.408233 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:35.408158    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 13:02:35.408292 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:35.408235    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a32a253a-ecaf-4f90-af1c-27a7e53b9629-metrics-certs podName:a32a253a-ecaf-4f90-af1c-27a7e53b9629 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:51.408224465 +0000 UTC m=+33.289459664 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a32a253a-ecaf-4f90-af1c-27a7e53b9629-metrics-certs") pod "network-metrics-daemon-mbncl" (UID: "a32a253a-ecaf-4f90-af1c-27a7e53b9629") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 13:02:35.508663 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:35.508636    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz5ls\" (UniqueName: \"kubernetes.io/projected/a07f5a09-920e-462f-ab6d-7a2ac53d199e-kube-api-access-gz5ls\") pod \"network-check-target-rxd99\" (UID: \"a07f5a09-920e-462f-ab6d-7a2ac53d199e\") " pod="openshift-network-diagnostics/network-check-target-rxd99"
Oct 30 13:02:35.508807 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:35.508791    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 13:02:35.508847 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:35.508811    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 13:02:35.508847 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:35.508821    2576 projected.go:196] Error preparing data for projected volume kube-api-access-gz5ls for pod openshift-network-diagnostics/network-check-target-rxd99: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 13:02:35.508906 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:35.508869    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a07f5a09-920e-462f-ab6d-7a2ac53d199e-kube-api-access-gz5ls podName:a07f5a09-920e-462f-ab6d-7a2ac53d199e nodeName:}" failed. No retries permitted until 2026-10-30 13:02:51.508855419 +0000 UTC m=+33.390090617 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gz5ls" (UniqueName: "kubernetes.io/projected/a07f5a09-920e-462f-ab6d-7a2ac53d199e-kube-api-access-gz5ls") pod "network-check-target-rxd99" (UID: "a07f5a09-920e-462f-ab6d-7a2ac53d199e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 13:02:35.633792 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:35.633613    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxd99"
Oct 30 13:02:35.633891 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:35.633691    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrfqz"
Oct 30 13:02:35.633938 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:35.633876    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rxd99" podUID="a07f5a09-920e-462f-ab6d-7a2ac53d199e"
Oct 30 13:02:35.633980 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:35.633935    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrfqz" podUID="10b774a1-1476-4f91-b6e5-d52b3f45700e"
Oct 30 13:02:35.633980 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:35.633701    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbncl"
Oct 30 13:02:35.634062 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:35.634022    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mbncl" podUID="a32a253a-ecaf-4f90-af1c-27a7e53b9629"
Oct 30 13:02:35.917306 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:35.917236    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tnm_3a675ac7-fbe1-449b-accf-bb0456ed0bd3/ovn-acl-logging/0.log"
Oct 30 13:02:35.917658 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:35.917537    2576 generic.go:356] "Generic (PLEG): container finished" podID="3a675ac7-fbe1-449b-accf-bb0456ed0bd3" containerID="b515d20ead38e58fcfaaa3b97b85952de70b11decbbdc1ca22cf2b35fe70670c" exitCode=1
Oct 30 13:02:35.917658 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:35.917611    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-67tnm" event={"ID":"3a675ac7-fbe1-449b-accf-bb0456ed0bd3","Type":"ContainerStarted","Data":"38a3ec25897de5e0f605083c1a81b7a5cfacb1f213813143d42aa8e144a3388f"}
Oct 30 13:02:35.917753 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:35.917700    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-67tnm" event={"ID":"3a675ac7-fbe1-449b-accf-bb0456ed0bd3","Type":"ContainerStarted","Data":"7fa110a51f9ebcdef4b2df46a9f6c51dfc98491b549cd8674f6df0cdc6abe3cb"}
Oct 30 13:02:35.917784 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:35.917755    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-67tnm" event={"ID":"3a675ac7-fbe1-449b-accf-bb0456ed0bd3","Type":"ContainerStarted","Data":"ca808349446671803838031993631b20d7f1d27542ec218843562df7f2b04316"}
Oct 30 13:02:35.917813 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:35.917804    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-67tnm" event={"ID":"3a675ac7-fbe1-449b-accf-bb0456ed0bd3","Type":"ContainerStarted","Data":"f575518aa8675c1de9ed595223ff5affe897fe9f213fd205956fd66343f4c8b3"}
Oct 30 13:02:35.917847 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:35.917835    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-67tnm" event={"ID":"3a675ac7-fbe1-449b-accf-bb0456ed0bd3","Type":"ContainerDied","Data":"b515d20ead38e58fcfaaa3b97b85952de70b11decbbdc1ca22cf2b35fe70670c"}
Oct 30 13:02:35.917883 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:35.917865    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-67tnm" event={"ID":"3a675ac7-fbe1-449b-accf-bb0456ed0bd3","Type":"ContainerStarted","Data":"97ff25d8dbe0be26407f157593588edcb713a5088c6a022d776aefe1418f5728"}
Oct 30 13:02:35.920632 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:35.920606    2576 generic.go:356] "Generic (PLEG): container finished" podID="183f23e7-ae80-445b-960c-fff9b410ba5b" containerID="a04266638b5538f4488b3a0a1d5050b776751dbab333940f5d123974e80382b6" exitCode=0
Oct 30 13:02:35.920738 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:35.920638    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lrqnm" event={"ID":"183f23e7-ae80-445b-960c-fff9b410ba5b","Type":"ContainerDied","Data":"a04266638b5538f4488b3a0a1d5050b776751dbab333940f5d123974e80382b6"}
Oct 30 13:02:36.513953 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:36.513912    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wpqvv"
Oct 30 13:02:36.514174 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:36.514044    2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 13:02:36.514536 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:36.514522    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wpqvv"
Oct 30 13:02:36.926517 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:36.926478    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lrqnm" event={"ID":"183f23e7-ae80-445b-960c-fff9b410ba5b","Type":"ContainerStarted","Data":"6f333be13ee5225e75d5241af4c593ccdc42d37ddc58858ed6f9c34c6884a88e"}
Oct 30 13:02:36.990249 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:36.990198    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lrqnm" podStartSLOduration=6.837515508 podStartE2EDuration="17.990184455s" totalImagesPullingTime="11.152668947s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:02:19 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:36.97872572 +0000 UTC m=+18.859960934" watchObservedRunningTime="2026-10-30 13:02:36.990184455 +0000 UTC m=+18.871419675"
Oct 30 13:02:37.633299 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:37.633263    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrfqz"
Oct 30 13:02:37.633472 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:37.633363    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbncl"
Oct 30 13:02:37.633472 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:37.633378    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrfqz" podUID="10b774a1-1476-4f91-b6e5-d52b3f45700e"
Oct 30 13:02:37.633472 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:37.633389    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxd99"
Oct 30 13:02:37.633472 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:37.633453    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mbncl" podUID="a32a253a-ecaf-4f90-af1c-27a7e53b9629"
Oct 30 13:02:37.633616 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:37.633506    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rxd99" podUID="a07f5a09-920e-462f-ab6d-7a2ac53d199e"
Oct 30 13:02:37.931917 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:37.931887    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tnm_3a675ac7-fbe1-449b-accf-bb0456ed0bd3/ovn-acl-logging/0.log"
Oct 30 13:02:37.932430 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:37.932403    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-67tnm" event={"ID":"3a675ac7-fbe1-449b-accf-bb0456ed0bd3","Type":"ContainerStarted","Data":"17c2321ce4bbe771ffa8854d44aaad0625b58687684aaa04553da7f49821b6c6"}
Oct 30 13:02:39.634104 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:39.634066    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxd99"
Oct 30 13:02:39.634518 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:39.634121    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrfqz"
Oct 30 13:02:39.634518 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:39.634081    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbncl"
Oct 30 13:02:39.634518 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:39.634216    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rxd99" podUID="a07f5a09-920e-462f-ab6d-7a2ac53d199e"
Oct 30 13:02:39.634518 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:39.634246    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrfqz" podUID="10b774a1-1476-4f91-b6e5-d52b3f45700e"
Oct 30 13:02:39.634518 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:39.634321    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mbncl" podUID="a32a253a-ecaf-4f90-af1c-27a7e53b9629"
Oct 30 13:02:40.944840 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:40.944669    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tnm_3a675ac7-fbe1-449b-accf-bb0456ed0bd3/ovn-acl-logging/0.log"
Oct 30 13:02:40.945450 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:40.945208    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-67tnm" event={"ID":"3a675ac7-fbe1-449b-accf-bb0456ed0bd3","Type":"ContainerStarted","Data":"fb17e226e24c1cdf61bfc4f6d9e07b8c43a450d0a2c31c172e57fae01ef6a3b8"}
Oct 30 13:02:40.945450 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:40.945434    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:40.945556 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:40.945496    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:40.945643 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:40.945617    2576 scope.go:122] "RemoveContainer" containerID="b515d20ead38e58fcfaaa3b97b85952de70b11decbbdc1ca22cf2b35fe70670c"
Oct 30 13:02:40.961701 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:40.961684    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:40.961783 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:40.961774    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:41.633272 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:41.633243    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxd99"
Oct 30 13:02:41.633422 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:41.633345    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rxd99" podUID="a07f5a09-920e-462f-ab6d-7a2ac53d199e"
Oct 30 13:02:41.633422 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:41.633259    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrfqz"
Oct 30 13:02:41.633422 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:41.633249    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbncl"
Oct 30 13:02:41.633538 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:41.633421    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrfqz" podUID="10b774a1-1476-4f91-b6e5-d52b3f45700e"
Oct 30 13:02:41.633538 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:41.633472    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mbncl" podUID="a32a253a-ecaf-4f90-af1c-27a7e53b9629"
Oct 30 13:02:41.937849 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:41.937816    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wrfqz"]
Oct 30 13:02:41.941085 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:41.941057    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rxd99"]
Oct 30 13:02:41.941939 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:41.941915    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mbncl"]
Oct 30 13:02:41.951044 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:41.951020    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tnm_3a675ac7-fbe1-449b-accf-bb0456ed0bd3/ovn-acl-logging/0.log"
Oct 30 13:02:41.951456 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:41.951438    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbncl"
Oct 30 13:02:41.951510 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:41.951467    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrfqz"
Oct 30 13:02:41.951510 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:41.951483    2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 13:02:41.951510 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:41.951483    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-67tnm" event={"ID":"3a675ac7-fbe1-449b-accf-bb0456ed0bd3","Type":"ContainerStarted","Data":"cb4a731c68b85bcffa75124b90d38089cd1a6c13ddbe5822247a2a43b255928f"}
Oct 30 13:02:41.951635 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:41.951549    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxd99"
Oct 30 13:02:41.951635 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:41.951565    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mbncl" podUID="a32a253a-ecaf-4f90-af1c-27a7e53b9629"
Oct 30 13:02:41.951729 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:41.951654    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrfqz" podUID="10b774a1-1476-4f91-b6e5-d52b3f45700e"
Oct 30 13:02:41.951777 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:41.951744    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rxd99" podUID="a07f5a09-920e-462f-ab6d-7a2ac53d199e"
Oct 30 13:02:42.953461 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:42.953432    2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 13:02:43.633734 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:43.633702    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrfqz"
Oct 30 13:02:43.633907 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:43.633779    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbncl"
Oct 30 13:02:43.633907 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:43.633864    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxd99"
Oct 30 13:02:43.633907 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:43.633888    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrfqz" podUID="10b774a1-1476-4f91-b6e5-d52b3f45700e"
Oct 30 13:02:43.634018 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:43.633932    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rxd99" podUID="a07f5a09-920e-462f-ab6d-7a2ac53d199e"
Oct 30 13:02:43.634048 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:43.634013    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mbncl" podUID="a32a253a-ecaf-4f90-af1c-27a7e53b9629"
Oct 30 13:02:45.633264 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:45.633230    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrfqz"
Oct 30 13:02:45.633264 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:45.633266    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxd99"
Oct 30 13:02:45.633895 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:45.633273    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbncl"
Oct 30 13:02:45.633895 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:45.633339    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrfqz" podUID="10b774a1-1476-4f91-b6e5-d52b3f45700e"
Oct 30 13:02:45.633895 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:45.633453    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rxd99" podUID="a07f5a09-920e-462f-ab6d-7a2ac53d199e"
Oct 30 13:02:45.633895 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:45.633552    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mbncl" podUID="a32a253a-ecaf-4f90-af1c-27a7e53b9629"
Oct 30 13:02:47.633364 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:47.633324    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxd99"
Oct 30 13:02:47.633364 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:47.633364    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbncl"
Oct 30 13:02:47.633852 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:47.633454    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rxd99" podUID="a07f5a09-920e-462f-ab6d-7a2ac53d199e"
Oct 30 13:02:47.633852 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:47.633504    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrfqz"
Oct 30 13:02:47.633852 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:47.633634    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mbncl" podUID="a32a253a-ecaf-4f90-af1c-27a7e53b9629"
Oct 30 13:02:47.633852 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:47.633740    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrfqz" podUID="10b774a1-1476-4f91-b6e5-d52b3f45700e"
Oct 30 13:02:48.357071 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:48.356821    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:48.357282 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:48.357267    2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 13:02:48.368246 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:48.368222    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:02:48.445885 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:48.445832    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-67tnm" podStartSLOduration=14.193059191 podStartE2EDuration="29.445818647s" totalImagesPullingTime="15.252759456s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:02:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:42.014074586 +0000 UTC m=+23.895309806" watchObservedRunningTime="2026-10-30 13:02:48.445818647 +0000 UTC m=+30.327053866"
Oct 30 13:02:49.398277 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.398202    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-31.ec2.internal" event="NodeReady"
Oct 30 13:02:49.398625 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.398322    2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 13:02:49.445023 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.444994    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-rbf7t"]
Oct 30 13:02:49.495944 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.495910    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-22fm4"]
Oct 30 13:02:49.496163 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.496048    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-rbf7t"
Oct 30 13:02:49.504971 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.504943    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wnzh2"]
Oct 30 13:02:49.505266 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.505239    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-22fm4"
Oct 30 13:02:49.512767 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.512745    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 13:02:49.512767 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.512758    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 13:02:49.513275 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.513254    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-lgssx"]
Oct 30 13:02:49.513401 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.513382    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-wnzh2"
Oct 30 13:02:49.516753 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.516723    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gxcrm"]
Oct 30 13:02:49.516873 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.516851    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-lgssx"
Oct 30 13:02:49.521065 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.521031    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5b97bd7c47-dssmx"]
Oct 30 13:02:49.521385 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.521366    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gxcrm"
Oct 30 13:02:49.532015 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.531991    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 13:02:49.532106 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.532091    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-9qjrr\""
Oct 30 13:02:49.534312 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.534293    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 13:02:49.534634 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.534620    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-mc5c8\""
Oct 30 13:02:49.536424 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.536404    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-668b54766c-wgk28"]
Oct 30 13:02:49.536679 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.536567    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:02:49.550304 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.550285    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 13:02:49.550386 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.550292    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9mrzf\""
Oct 30 13:02:49.550386 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.550305    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 13:02:49.550386 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.550353    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 13:02:49.560269 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.560251    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lt2qg"]
Oct 30 13:02:49.560371 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.560357    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-668b54766c-wgk28"
Oct 30 13:02:49.579427 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.579408    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-9677cf66b-kpdfl"]
Oct 30 13:02:49.579544 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.579528    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lt2qg"
Oct 30 13:02:49.595512 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.595491    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-lvf7h"]
Oct 30 13:02:49.595615 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.595601    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:02:49.611287 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.611264    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7cca8a39-bee0-4705-bcde-b31060e44944-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gxcrm\" (UID: \"7cca8a39-bee0-4705-bcde-b31060e44944\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gxcrm"
Oct 30 13:02:49.611385 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.611295    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-262nq\" (UniqueName: \"kubernetes.io/projected/8021bc0b-4447-4f04-b9bd-5f80e9cf3507-kube-api-access-262nq\") pod \"kas-connection-checker-685895c7c9-rbf7t\" (UID: \"8021bc0b-4447-4f04-b9bd-5f80e9cf3507\") " pod="kube-system/kas-connection-checker-685895c7c9-rbf7t"
Oct 30 13:02:49.611385 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.611311    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/155bbdd4-b566-467f-88c6-ec5b4b881a09-serving-cert\") pod \"insights-operator-7c4986d449-lgssx\" (UID: \"155bbdd4-b566-467f-88c6-ec5b4b881a09\") " pod="openshift-insights/insights-operator-7c4986d449-lgssx"
Oct 30 13:02:49.611385 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.611341    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsj7z\" (UniqueName: \"kubernetes.io/projected/155bbdd4-b566-467f-88c6-ec5b4b881a09-kube-api-access-lsj7z\") pod \"insights-operator-7c4986d449-lgssx\" (UID: \"155bbdd4-b566-467f-88c6-ec5b4b881a09\") " pod="openshift-insights/insights-operator-7c4986d449-lgssx"
Oct 30 13:02:49.611522 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.611405    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbpg6\" (UniqueName: \"kubernetes.io/projected/46bedd0c-7237-4128-be98-69e4474bb535-kube-api-access-kbpg6\") pod \"kas-connection-checker-685895c7c9-22fm4\" (UID: \"46bedd0c-7237-4128-be98-69e4474bb535\") " pod="kube-system/kas-connection-checker-685895c7c9-22fm4"
Oct 30 13:02:49.611559 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.611518    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/7cca8a39-bee0-4705-bcde-b31060e44944-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-gxcrm\" (UID: \"7cca8a39-bee0-4705-bcde-b31060e44944\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gxcrm"
Oct 30 13:02:49.611559 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.611547    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kncj9\" (UniqueName: \"kubernetes.io/projected/7cca8a39-bee0-4705-bcde-b31060e44944-kube-api-access-kncj9\") pod \"cluster-samples-operator-6ddb6c8b94-gxcrm\" (UID: \"7cca8a39-bee0-4705-bcde-b31060e44944\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gxcrm"
Oct 30 13:02:49.611628 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.611597    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/155bbdd4-b566-467f-88c6-ec5b4b881a09-service-ca-bundle\") pod \"insights-operator-7c4986d449-lgssx\" (UID: \"155bbdd4-b566-467f-88c6-ec5b4b881a09\") " pod="openshift-insights/insights-operator-7c4986d449-lgssx"
Oct 30 13:02:49.611628 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.611622    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjjr5\" (UniqueName: \"kubernetes.io/projected/44f513ab-9115-4761-aa8b-c546212ec17b-kube-api-access-vjjr5\") pod \"kas-connection-checker-685895c7c9-wnzh2\" (UID: \"44f513ab-9115-4761-aa8b-c546212ec17b\") " pod="kube-system/kas-connection-checker-685895c7c9-wnzh2"
Oct 30 13:02:49.611712 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.611680    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/155bbdd4-b566-467f-88c6-ec5b4b881a09-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-lgssx\" (UID: \"155bbdd4-b566-467f-88c6-ec5b4b881a09\") " pod="openshift-insights/insights-operator-7c4986d449-lgssx"
Oct 30 13:02:49.611746 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.611723    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/155bbdd4-b566-467f-88c6-ec5b4b881a09-tmp\") pod \"insights-operator-7c4986d449-lgssx\" (UID: \"155bbdd4-b566-467f-88c6-ec5b4b881a09\") " pod="openshift-insights/insights-operator-7c4986d449-lgssx"
Oct 30 13:02:49.611781 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.611751    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/155bbdd4-b566-467f-88c6-ec5b4b881a09-snapshots\") pod \"insights-operator-7c4986d449-lgssx\" (UID: \"155bbdd4-b566-467f-88c6-ec5b4b881a09\") " pod="openshift-insights/insights-operator-7c4986d449-lgssx"
Oct 30 13:02:49.612026 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.612012    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 13:02:49.612110 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.612097    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 13:02:49.612171 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.612100    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 13:02:49.612171 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.612157    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 13:02:49.612250 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.612101    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 13:02:49.612299 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.612249    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 13:02:49.612299 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.612254    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 13:02:49.612299 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.612276    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 13:02:49.612299 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.612282    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 13:02:49.612451 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.612437    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 13:02:49.612486 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.612461    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 13:02:49.612541 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.612529    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 13:02:49.615355 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.615336    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-6jdj4"]
Oct 30 13:02:49.615463 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.615448    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-lvf7h"
Oct 30 13:02:49.625573 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.625552    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 13:02:49.625756 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.625740    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 13:02:49.634562 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.634543    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 13:02:49.634684 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.634669    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-jp6bq\""
Oct 30 13:02:49.634737 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.634685    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 13:02:49.634737 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.634681    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 13:02:49.635067 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.635051    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-6cbwz\""
Oct 30 13:02:49.635116 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.635104    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 13:02:49.635187 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.635173    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-fsk4s\""
Oct 30 13:02:49.637499 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.637485    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-x8wqt"]
Oct 30 13:02:49.637582 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.637572    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxd99"
Oct 30 13:02:49.637677 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.637661    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbncl"
Oct 30 13:02:49.637794 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.637778    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrfqz"
Oct 30 13:02:49.637960 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.637944    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-6jdj4"
Oct 30 13:02:49.653620 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.653595    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-fhnzt"]
Oct 30 13:02:49.653715 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.653700    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x8wqt"
Oct 30 13:02:49.667699 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.667680    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-2w8jt"]
Oct 30 13:02:49.667794 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.667780    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fhnzt"
Oct 30 13:02:49.686270 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.686246    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-r7vrw"]
Oct 30 13:02:49.686368 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.686354    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-2w8jt"
Oct 30 13:02:49.700429 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.700409    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8nb7s"]
Oct 30 13:02:49.700527 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.700513    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-r7vrw"
Oct 30 13:02:49.712002 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.711980    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-trusted-ca\") pod \"image-registry-9677cf66b-kpdfl\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") " pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:02:49.712104 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.712008    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdrpm\" (UniqueName: \"kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-kube-api-access-vdrpm\") pod \"image-registry-9677cf66b-kpdfl\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") " pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:02:49.712104 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.712032    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/155bbdd4-b566-467f-88c6-ec5b4b881a09-tmp\") pod \"insights-operator-7c4986d449-lgssx\" (UID: \"155bbdd4-b566-467f-88c6-ec5b4b881a09\") " pod="openshift-insights/insights-operator-7c4986d449-lgssx"
Oct 30 13:02:49.712104 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.712050    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/155bbdd4-b566-467f-88c6-ec5b4b881a09-snapshots\") pod \"insights-operator-7c4986d449-lgssx\" (UID: \"155bbdd4-b566-467f-88c6-ec5b4b881a09\") " pod="openshift-insights/insights-operator-7c4986d449-lgssx"
Oct 30 13:02:49.712104 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.712099    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 13:02:49.712336 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.712146    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 13:02:49.712336 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.712202    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7cca8a39-bee0-4705-bcde-b31060e44944-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gxcrm\" (UID: \"7cca8a39-bee0-4705-bcde-b31060e44944\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gxcrm"
Oct 30 13:02:49.712336 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.712207    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 13:02:49.712336 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:49.712291    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 13:02:49.712336 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.712326    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/feda4324-4a5f-491b-a04c-202c222261a0-trusted-ca\") pod \"image-registry-5b97bd7c47-dssmx\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") " pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:02:49.712567 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.712360    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 13:02:49.712567 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.712392    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 13:02:49.712567 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:49.712361    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cca8a39-bee0-4705-bcde-b31060e44944-samples-operator-tls podName:7cca8a39-bee0-4705-bcde-b31060e44944 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:50.212343096 +0000 UTC m=+32.093578312 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7cca8a39-bee0-4705-bcde-b31060e44944-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-gxcrm" (UID: "7cca8a39-bee0-4705-bcde-b31060e44944") : secret "samples-operator-tls" not found
Oct 30 13:02:49.712567 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.712427    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-default-certificate\") pod \"router-default-668b54766c-wgk28\" (UID: \"d5db3ba8-f1de-4fa5-9839-e7e8200da58f\") " pod="openshift-ingress/router-default-668b54766c-wgk28"
Oct 30 13:02:49.712567 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.712458    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-registry-certificates\") pod \"image-registry-9677cf66b-kpdfl\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") " pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:02:49.712567 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.712469    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 13:02:49.712567 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.712453    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/155bbdd4-b566-467f-88c6-ec5b4b881a09-tmp\") pod \"insights-operator-7c4986d449-lgssx\" (UID: \"155bbdd4-b566-467f-88c6-ec5b4b881a09\") " pod="openshift-insights/insights-operator-7c4986d449-lgssx"
Oct 30 13:02:49.712567 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.712498    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/155bbdd4-b566-467f-88c6-ec5b4b881a09-serving-cert\") pod \"insights-operator-7c4986d449-lgssx\" (UID: \"155bbdd4-b566-467f-88c6-ec5b4b881a09\") " pod="openshift-insights/insights-operator-7c4986d449-lgssx"
Oct 30 13:02:49.712567 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.712511    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 13:02:49.712567 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.712527    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 13:02:49.712567 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.712564    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 13:02:49.712567 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.712566    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-ca-trust-extracted\") pod \"image-registry-9677cf66b-kpdfl\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") " pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:02:49.713080 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.712581    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 13:02:49.713080 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.712618    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsj7z\" (UniqueName: \"kubernetes.io/projected/155bbdd4-b566-467f-88c6-ec5b4b881a09-kube-api-access-lsj7z\") pod \"insights-operator-7c4986d449-lgssx\" (UID: \"155bbdd4-b566-467f-88c6-ec5b4b881a09\") " pod="openshift-insights/insights-operator-7c4986d449-lgssx"
Oct 30 13:02:49.713080 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.712646    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbpg6\" (UniqueName: \"kubernetes.io/projected/46bedd0c-7237-4128-be98-69e4474bb535-kube-api-access-kbpg6\") pod \"kas-connection-checker-685895c7c9-22fm4\" (UID: \"46bedd0c-7237-4128-be98-69e4474bb535\") " pod="kube-system/kas-connection-checker-685895c7c9-22fm4"
Oct 30 13:02:49.713080 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.712687    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5cbda802-f819-4ea6-846e-440cea4afed2-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lt2qg\" (UID: \"5cbda802-f819-4ea6-846e-440cea4afed2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lt2qg"
Oct 30 13:02:49.713080 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.712717    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncqh7\" (UniqueName: \"kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-kube-api-access-ncqh7\") pod \"image-registry-5b97bd7c47-dssmx\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") " pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:02:49.713080 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.712723    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 13:02:49.713080 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.712683    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 13:02:49.713080 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.712729    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/155bbdd4-b566-467f-88c6-ec5b4b881a09-snapshots\") pod \"insights-operator-7c4986d449-lgssx\" (UID: \"155bbdd4-b566-467f-88c6-ec5b4b881a09\") " pod="openshift-insights/insights-operator-7c4986d449-lgssx"
Oct 30 13:02:49.713080 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.712757    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-bound-sa-token\") pod \"image-registry-9677cf66b-kpdfl\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") " pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:02:49.713080 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.712801    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/7cca8a39-bee0-4705-bcde-b31060e44944-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-gxcrm\" (UID: \"7cca8a39-bee0-4705-bcde-b31060e44944\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gxcrm"
Oct 30 13:02:49.713080 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.712838    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kncj9\" (UniqueName: \"kubernetes.io/projected/7cca8a39-bee0-4705-bcde-b31060e44944-kube-api-access-kncj9\") pod \"cluster-samples-operator-6ddb6c8b94-gxcrm\" (UID: \"7cca8a39-bee0-4705-bcde-b31060e44944\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gxcrm"
Oct 30 13:02:49.713080 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.712868    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-bound-sa-token\") pod \"image-registry-5b97bd7c47-dssmx\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") " pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:02:49.713080 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.712894    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-service-ca-bundle\") pod \"router-default-668b54766c-wgk28\" (UID: \"d5db3ba8-f1de-4fa5-9839-e7e8200da58f\") " pod="openshift-ingress/router-default-668b54766c-wgk28"
Oct 30 13:02:49.713080 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.712929    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 13:02:49.713080 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.712975    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-registry-tls\") pod \"image-registry-5b97bd7c47-dssmx\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") " pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:02:49.713080 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.713002    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/155bbdd4-b566-467f-88c6-ec5b4b881a09-service-ca-bundle\") pod \"insights-operator-7c4986d449-lgssx\" (UID: \"155bbdd4-b566-467f-88c6-ec5b4b881a09\") " pod="openshift-insights/insights-operator-7c4986d449-lgssx"
Oct 30 13:02:49.713080 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.713034    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-metrics-certs\") pod \"router-default-668b54766c-wgk28\" (UID: \"d5db3ba8-f1de-4fa5-9839-e7e8200da58f\") " pod="openshift-ingress/router-default-668b54766c-wgk28"
Oct 30 13:02:49.713785 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.713073    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/155bbdd4-b566-467f-88c6-ec5b4b881a09-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-lgssx\" (UID: \"155bbdd4-b566-467f-88c6-ec5b4b881a09\") " pod="openshift-insights/insights-operator-7c4986d449-lgssx"
Oct 30 13:02:49.713785 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.713096    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 13:02:49.713785 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.713114    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-installation-pull-secrets\") pod \"image-registry-9677cf66b-kpdfl\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") " pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:02:49.713785 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.713219    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/feda4324-4a5f-491b-a04c-202c222261a0-image-registry-private-configuration\") pod \"image-registry-5b97bd7c47-dssmx\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") " pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:02:49.713785 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.713246    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-stats-auth\") pod \"router-default-668b54766c-wgk28\" (UID: \"d5db3ba8-f1de-4fa5-9839-e7e8200da58f\") " pod="openshift-ingress/router-default-668b54766c-wgk28"
Oct 30 13:02:49.713785 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.713280    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-262nq\" (UniqueName: \"kubernetes.io/projected/8021bc0b-4447-4f04-b9bd-5f80e9cf3507-kube-api-access-262nq\") pod \"kas-connection-checker-685895c7c9-rbf7t\" (UID: \"8021bc0b-4447-4f04-b9bd-5f80e9cf3507\") " pod="kube-system/kas-connection-checker-685895c7c9-rbf7t"
Oct 30 13:02:49.713785 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.713305    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/feda4324-4a5f-491b-a04c-202c222261a0-ca-trust-extracted\") pod \"image-registry-5b97bd7c47-dssmx\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") " pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:02:49.713785 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.713328    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5cbda802-f819-4ea6-846e-440cea4afed2-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lt2qg\" (UID: \"5cbda802-f819-4ea6-846e-440cea4afed2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lt2qg"
Oct 30 13:02:49.713785 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.713357    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-registry-tls\") pod \"image-registry-9677cf66b-kpdfl\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") " pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:02:49.713785 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.713382    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfxrl\" (UniqueName: \"kubernetes.io/projected/abf08e3d-eec3-4f73-92a6-4ffb691cbd2d-kube-api-access-tfxrl\") pod \"volume-data-source-validator-58c9f5f46d-lvf7h\" (UID: \"abf08e3d-eec3-4f73-92a6-4ffb691cbd2d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-lvf7h"
Oct 30 13:02:49.713785 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.713413    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/feda4324-4a5f-491b-a04c-202c222261a0-registry-certificates\") pod \"image-registry-5b97bd7c47-dssmx\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") " pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:02:49.714244 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.714166    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/7cca8a39-bee0-4705-bcde-b31060e44944-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-gxcrm\" (UID: \"7cca8a39-bee0-4705-bcde-b31060e44944\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gxcrm"
Oct 30 13:02:49.714289 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.714263    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/155bbdd4-b566-467f-88c6-ec5b4b881a09-service-ca-bundle\") pod \"insights-operator-7c4986d449-lgssx\" (UID: \"155bbdd4-b566-467f-88c6-ec5b4b881a09\") " pod="openshift-insights/insights-operator-7c4986d449-lgssx"
Oct 30 13:02:49.714611 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.714588    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-image-registry-private-configuration\") pod \"image-registry-9677cf66b-kpdfl\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") " pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:02:49.714683 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.714637    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rflhr\" (UniqueName: \"kubernetes.io/projected/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-kube-api-access-rflhr\") pod \"router-default-668b54766c-wgk28\" (UID: \"d5db3ba8-f1de-4fa5-9839-e7e8200da58f\") " pod="openshift-ingress/router-default-668b54766c-wgk28"
Oct 30 13:02:49.714683 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.714668    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/155bbdd4-b566-467f-88c6-ec5b4b881a09-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-lgssx\" (UID: \"155bbdd4-b566-467f-88c6-ec5b4b881a09\") " pod="openshift-insights/insights-operator-7c4986d449-lgssx"
Oct 30 13:02:49.714769 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.714743    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjjr5\" (UniqueName: \"kubernetes.io/projected/44f513ab-9115-4761-aa8b-c546212ec17b-kube-api-access-vjjr5\") pod \"kas-connection-checker-685895c7c9-wnzh2\" (UID: \"44f513ab-9115-4761-aa8b-c546212ec17b\") " pod="kube-system/kas-connection-checker-685895c7c9-wnzh2"
Oct 30 13:02:49.714817 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.714780    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48dsv\" (UniqueName: \"kubernetes.io/projected/5cbda802-f819-4ea6-846e-440cea4afed2-kube-api-access-48dsv\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lt2qg\" (UID: \"5cbda802-f819-4ea6-846e-440cea4afed2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lt2qg"
Oct 30 13:02:49.714959 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.714812    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/feda4324-4a5f-491b-a04c-202c222261a0-installation-pull-secrets\") pod \"image-registry-5b97bd7c47-dssmx\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") " pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:02:49.716191 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.715214    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-rbf7t"]
Oct 30 13:02:49.716191 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.715263    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wnzh2"]
Oct 30 13:02:49.716191 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.715285    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-22fm4"]
Oct 30 13:02:49.716191 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.715314    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wmb4b"]
Oct 30 13:02:49.716191 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.715453    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8nb7s"
Oct 30 13:02:49.717963 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.717930    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/155bbdd4-b566-467f-88c6-ec5b4b881a09-serving-cert\") pod \"insights-operator-7c4986d449-lgssx\" (UID: \"155bbdd4-b566-467f-88c6-ec5b4b881a09\") " pod="openshift-insights/insights-operator-7c4986d449-lgssx"
Oct 30 13:02:49.724295 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.724272    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 13:02:49.731725 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.731699    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsj7z\" (UniqueName: \"kubernetes.io/projected/155bbdd4-b566-467f-88c6-ec5b4b881a09-kube-api-access-lsj7z\") pod \"insights-operator-7c4986d449-lgssx\" (UID: \"155bbdd4-b566-467f-88c6-ec5b4b881a09\") " pod="openshift-insights/insights-operator-7c4986d449-lgssx"
Oct 30 13:02:49.731821 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.731758    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbpg6\" (UniqueName: \"kubernetes.io/projected/46bedd0c-7237-4128-be98-69e4474bb535-kube-api-access-kbpg6\") pod \"kas-connection-checker-685895c7c9-22fm4\" (UID: \"46bedd0c-7237-4128-be98-69e4474bb535\") " pod="kube-system/kas-connection-checker-685895c7c9-22fm4"
Oct 30 13:02:49.732042 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.732025    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-262nq\" (UniqueName: \"kubernetes.io/projected/8021bc0b-4447-4f04-b9bd-5f80e9cf3507-kube-api-access-262nq\") pod \"kas-connection-checker-685895c7c9-rbf7t\" (UID: \"8021bc0b-4447-4f04-b9bd-5f80e9cf3507\") " pod="kube-system/kas-connection-checker-685895c7c9-rbf7t"
Oct 30 13:02:49.732368 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.732349    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kncj9\" (UniqueName: \"kubernetes.io/projected/7cca8a39-bee0-4705-bcde-b31060e44944-kube-api-access-kncj9\") pod \"cluster-samples-operator-6ddb6c8b94-gxcrm\" (UID: \"7cca8a39-bee0-4705-bcde-b31060e44944\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gxcrm"
Oct 30 13:02:49.732740 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.732722    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjjr5\" (UniqueName: \"kubernetes.io/projected/44f513ab-9115-4761-aa8b-c546212ec17b-kube-api-access-vjjr5\") pod \"kas-connection-checker-685895c7c9-wnzh2\" (UID: \"44f513ab-9115-4761-aa8b-c546212ec17b\") " pod="kube-system/kas-connection-checker-685895c7c9-wnzh2"
Oct 30 13:02:49.733208 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.733188    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 13:02:49.733344 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.733200    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 13:02:49.733409 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.733222    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-d86xq\""
Oct 30 13:02:49.733409 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.733221    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 13:02:49.733409 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.733224    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 13:02:49.733409 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.733224    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-k8kgp\""
Oct 30 13:02:49.733409 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.733276    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 13:02:49.733633 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.733414    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tdfvc\""
Oct 30 13:02:49.733633 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.733420    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 13:02:49.733633 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.733440    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-tzhgp\""
Oct 30 13:02:49.733633 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.733457    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-5crmk\""
Oct 30 13:02:49.733633 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.733544    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tzck4\""
Oct 30 13:02:49.733633 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.733544    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9hcn4\""
Oct 30 13:02:49.733633 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.733569    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-9xcbx\""
Oct 30 13:02:49.733953 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.733637    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-76hdh\""
Oct 30 13:02:49.733953 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.733655    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 13:02:49.735700 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.735684    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-lgssx"]
Oct 30 13:02:49.735777 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.735709    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lt2qg"]
Oct 30 13:02:49.735777 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.735725    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b97bd7c47-dssmx"]
Oct 30 13:02:49.735777 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.735743    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-668b54766c-wgk28"]
Oct 30 13:02:49.735917 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.735833    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-r7vrw"]
Oct 30 13:02:49.735917 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.735866    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wmb4b"
Oct 30 13:02:49.735917 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.735903    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8nb7s"]
Oct 30 13:02:49.736057 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.735984    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-fhnzt"]
Oct 30 13:02:49.736057 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.736051    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gxcrm"]
Oct 30 13:02:49.736175 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.736083    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-lvf7h"]
Oct 30 13:02:49.736175 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.736105    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-9677cf66b-kpdfl"]
Oct 30 13:02:49.736175 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.736120    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-6jdj4"]
Oct 30 13:02:49.736175 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.736157    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-x8wqt"]
Oct 30 13:02:49.736366 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.736181    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-2w8jt"]
Oct 30 13:02:49.736366 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.736203    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wmb4b"]
Oct 30 13:02:49.749902 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.749871    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-hwzrd\""
Oct 30 13:02:49.749902 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.749876    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 13:02:49.811994 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.811943    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 13:02:49.812110 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.812058    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 13:02:49.812110 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.812066    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 13:02:49.815211 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.815199    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-rbf7t"
Oct 30 13:02:49.815331 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.815316    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncqh7\" (UniqueName: \"kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-kube-api-access-ncqh7\") pod \"image-registry-5b97bd7c47-dssmx\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") " pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:02:49.815374 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.815340    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-bound-sa-token\") pod \"image-registry-9677cf66b-kpdfl\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") " pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:02:49.815374 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.815362    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d0b2655-d52d-4c95-9589-d45237fa1057-tmp-dir\") pod \"dns-default-8nb7s\" (UID: \"0d0b2655-d52d-4c95-9589-d45237fa1057\") " pod="openshift-dns/dns-default-8nb7s"
Oct 30 13:02:49.815454 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.815382    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r598w\" (UniqueName: \"kubernetes.io/projected/e640effb-1ff4-40f2-8d7b-53732edcd2bf-kube-api-access-r598w\") pod \"cluster-monitoring-operator-884576c57-x8wqt\" (UID: \"e640effb-1ff4-40f2-8d7b-53732edcd2bf\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x8wqt"
Oct 30 13:02:49.815454 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.815400    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-bound-sa-token\") pod \"image-registry-5b97bd7c47-dssmx\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") " pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:02:49.815454 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.815416    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-service-ca-bundle\") pod \"router-default-668b54766c-wgk28\" (UID: \"d5db3ba8-f1de-4fa5-9839-e7e8200da58f\") " pod="openshift-ingress/router-default-668b54766c-wgk28"
Oct 30 13:02:49.815540 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.815463    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f93ee017-ab79-447c-90e0-5b85b71dba24-config\") pod \"console-operator-7cd56b8dbc-6jdj4\" (UID: \"f93ee017-ab79-447c-90e0-5b85b71dba24\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6jdj4"
Oct 30 13:02:49.815540 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.815494    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-registry-tls\") pod \"image-registry-5b97bd7c47-dssmx\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") " pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:02:49.815540 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.815520    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-metrics-certs\") pod \"router-default-668b54766c-wgk28\" (UID: \"d5db3ba8-f1de-4fa5-9839-e7e8200da58f\") " pod="openshift-ingress/router-default-668b54766c-wgk28"
Oct 30 13:02:49.815619 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.815543    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f93ee017-ab79-447c-90e0-5b85b71dba24-serving-cert\") pod \"console-operator-7cd56b8dbc-6jdj4\" (UID: \"f93ee017-ab79-447c-90e0-5b85b71dba24\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6jdj4"
Oct 30 13:02:49.815619 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.815566    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnx68\" (UniqueName: \"kubernetes.io/projected/4ad66a0c-8dc0-40a4-a1b2-28d26b0de94b-kube-api-access-rnx68\") pod \"network-check-source-6f78d59f99-r7vrw\" (UID: \"4ad66a0c-8dc0-40a4-a1b2-28d26b0de94b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-r7vrw"
Oct 30 13:02:49.815619 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.815593    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-installation-pull-secrets\") pod \"image-registry-9677cf66b-kpdfl\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") " pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:02:49.815748 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.815624    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/feda4324-4a5f-491b-a04c-202c222261a0-image-registry-private-configuration\") pod \"image-registry-5b97bd7c47-dssmx\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") " pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:02:49.815748 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.815647    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-stats-auth\") pod \"router-default-668b54766c-wgk28\" (UID: \"d5db3ba8-f1de-4fa5-9839-e7e8200da58f\") " pod="openshift-ingress/router-default-668b54766c-wgk28"
Oct 30 13:02:49.815748 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:49.815695    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-service-ca-bundle podName:d5db3ba8-f1de-4fa5-9839-e7e8200da58f nodeName:}" failed. No retries permitted until 2026-10-30 13:02:50.3156702 +0000 UTC m=+32.196905412 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-service-ca-bundle") pod "router-default-668b54766c-wgk28" (UID: "d5db3ba8-f1de-4fa5-9839-e7e8200da58f") : configmap references non-existent config key: service-ca.crt
Oct 30 13:02:49.815748 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.815727    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plb6z\" (UniqueName: \"kubernetes.io/projected/0d0b2655-d52d-4c95-9589-d45237fa1057-kube-api-access-plb6z\") pod \"dns-default-8nb7s\" (UID: \"0d0b2655-d52d-4c95-9589-d45237fa1057\") " pod="openshift-dns/dns-default-8nb7s"
Oct 30 13:02:49.815923 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:49.815834    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 13:02:49.815923 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:49.815849    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b97bd7c47-dssmx: secret "image-registry-tls" not found
Oct 30 13:02:49.815923 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.815894    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/63086243-e82d-4fc5-bfef-e4241c22ad4e-nginx-conf\") pod \"networking-console-plugin-86c478bb88-2w8jt\" (UID: \"63086243-e82d-4fc5-bfef-e4241c22ad4e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2w8jt"
Oct 30 13:02:49.816071 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:49.815939    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-registry-tls podName:feda4324-4a5f-491b-a04c-202c222261a0 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:50.315920117 +0000 UTC m=+32.197155332 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-registry-tls") pod "image-registry-5b97bd7c47-dssmx" (UID: "feda4324-4a5f-491b-a04c-202c222261a0") : secret "image-registry-tls" not found
Oct 30 13:02:49.816071 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:49.815964    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 13:02:49.816071 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.816003    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/feda4324-4a5f-491b-a04c-202c222261a0-ca-trust-extracted\") pod \"image-registry-5b97bd7c47-dssmx\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") " pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:02:49.816071 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.816058    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5cbda802-f819-4ea6-846e-440cea4afed2-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lt2qg\" (UID: \"5cbda802-f819-4ea6-846e-440cea4afed2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lt2qg"
Oct 30 13:02:49.816071 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:49.816069    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-metrics-certs podName:d5db3ba8-f1de-4fa5-9839-e7e8200da58f nodeName:}" failed. No retries permitted until 2026-10-30 13:02:50.316057079 +0000 UTC m=+32.197292321 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-metrics-certs") pod "router-default-668b54766c-wgk28" (UID: "d5db3ba8-f1de-4fa5-9839-e7e8200da58f") : secret "router-metrics-certs-default" not found
Oct 30 13:02:49.816347 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.816103    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0d0b2655-d52d-4c95-9589-d45237fa1057-config-volume\") pod \"dns-default-8nb7s\" (UID: \"0d0b2655-d52d-4c95-9589-d45237fa1057\") " pod="openshift-dns/dns-default-8nb7s"
Oct 30 13:02:49.816347 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.816147    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-registry-tls\") pod \"image-registry-9677cf66b-kpdfl\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") " pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:02:49.816347 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.816176    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfxrl\" (UniqueName: \"kubernetes.io/projected/abf08e3d-eec3-4f73-92a6-4ffb691cbd2d-kube-api-access-tfxrl\") pod \"volume-data-source-validator-58c9f5f46d-lvf7h\" (UID: \"abf08e3d-eec3-4f73-92a6-4ffb691cbd2d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-lvf7h"
Oct 30 13:02:49.816347 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.816198    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/feda4324-4a5f-491b-a04c-202c222261a0-registry-certificates\") pod \"image-registry-5b97bd7c47-dssmx\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") " pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:02:49.816347 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.816215    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/79664154-5454-4294-baad-12bdfad6fe68-config\") pod \"service-ca-operator-544d749c8c-fhnzt\" (UID: \"79664154-5454-4294-baad-12bdfad6fe68\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fhnzt"
Oct 30 13:02:49.816347 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.816232    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f93ee017-ab79-447c-90e0-5b85b71dba24-trusted-ca\") pod \"console-operator-7cd56b8dbc-6jdj4\" (UID: \"f93ee017-ab79-447c-90e0-5b85b71dba24\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6jdj4"
Oct 30 13:02:49.816347 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.816258    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-image-registry-private-configuration\") pod \"image-registry-9677cf66b-kpdfl\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") " pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:02:49.816347 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.816282    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rflhr\" (UniqueName: \"kubernetes.io/projected/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-kube-api-access-rflhr\") pod \"router-default-668b54766c-wgk28\" (UID: \"d5db3ba8-f1de-4fa5-9839-e7e8200da58f\") " pod="openshift-ingress/router-default-668b54766c-wgk28"
Oct 30 13:02:49.816347 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.816298    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lkrg\" (UniqueName: \"kubernetes.io/projected/f93ee017-ab79-447c-90e0-5b85b71dba24-kube-api-access-2lkrg\") pod \"console-operator-7cd56b8dbc-6jdj4\" (UID: \"f93ee017-ab79-447c-90e0-5b85b71dba24\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6jdj4"
Oct 30 13:02:49.816347 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.816315    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d0b2655-d52d-4c95-9589-d45237fa1057-metrics-tls\") pod \"dns-default-8nb7s\" (UID: \"0d0b2655-d52d-4c95-9589-d45237fa1057\") " pod="openshift-dns/dns-default-8nb7s"
Oct 30 13:02:49.816347 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.816329    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk568\" (UniqueName: \"kubernetes.io/projected/1a25d5a7-83d4-443b-98ae-fc7b603cfc5f-kube-api-access-qk568\") pod \"ingress-canary-wmb4b\" (UID: \"1a25d5a7-83d4-443b-98ae-fc7b603cfc5f\") " pod="openshift-ingress-canary/ingress-canary-wmb4b"
Oct 30 13:02:49.816347 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.816344    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nrlb\" (UniqueName: \"kubernetes.io/projected/79664154-5454-4294-baad-12bdfad6fe68-kube-api-access-7nrlb\") pod \"service-ca-operator-544d749c8c-fhnzt\" (UID: \"79664154-5454-4294-baad-12bdfad6fe68\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fhnzt"
Oct 30 13:02:49.816922 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.816370    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48dsv\" (UniqueName: \"kubernetes.io/projected/5cbda802-f819-4ea6-846e-440cea4afed2-kube-api-access-48dsv\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lt2qg\" (UID: \"5cbda802-f819-4ea6-846e-440cea4afed2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lt2qg"
Oct 30 13:02:49.816922 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.816395    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/79664154-5454-4294-baad-12bdfad6fe68-serving-cert\") pod \"service-ca-operator-544d749c8c-fhnzt\" (UID: \"79664154-5454-4294-baad-12bdfad6fe68\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fhnzt"
Oct 30 13:02:49.816922 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.816419    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/feda4324-4a5f-491b-a04c-202c222261a0-installation-pull-secrets\") pod \"image-registry-5b97bd7c47-dssmx\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") " pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:02:49.816922 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.816445    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-trusted-ca\") pod \"image-registry-9677cf66b-kpdfl\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") " pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:02:49.816922 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.816460    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdrpm\" (UniqueName: \"kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-kube-api-access-vdrpm\") pod \"image-registry-9677cf66b-kpdfl\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") " pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:02:49.816922 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.816477    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1a25d5a7-83d4-443b-98ae-fc7b603cfc5f-cert\") pod \"ingress-canary-wmb4b\" (UID: \"1a25d5a7-83d4-443b-98ae-fc7b603cfc5f\") " pod="openshift-ingress-canary/ingress-canary-wmb4b"
Oct 30 13:02:49.816922 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.816511    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/feda4324-4a5f-491b-a04c-202c222261a0-trusted-ca\") pod \"image-registry-5b97bd7c47-dssmx\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") " pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:02:49.816922 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.816526    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-default-certificate\") pod \"router-default-668b54766c-wgk28\" (UID: \"d5db3ba8-f1de-4fa5-9839-e7e8200da58f\") " pod="openshift-ingress/router-default-668b54766c-wgk28"
Oct 30 13:02:49.816922 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.816541    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-registry-certificates\") pod \"image-registry-9677cf66b-kpdfl\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") " pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:02:49.816922 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.816563    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-ca-trust-extracted\") pod \"image-registry-9677cf66b-kpdfl\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") " pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:02:49.816922 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.816578    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e640effb-1ff4-40f2-8d7b-53732edcd2bf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-x8wqt\" (UID: \"e640effb-1ff4-40f2-8d7b-53732edcd2bf\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x8wqt"
Oct 30 13:02:49.816922 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.816623    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e640effb-1ff4-40f2-8d7b-53732edcd2bf-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-x8wqt\" (UID: \"e640effb-1ff4-40f2-8d7b-53732edcd2bf\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x8wqt"
Oct 30 13:02:49.816922 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.816622    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5cbda802-f819-4ea6-846e-440cea4afed2-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lt2qg\" (UID: \"5cbda802-f819-4ea6-846e-440cea4afed2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lt2qg"
Oct 30 13:02:49.816922 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.816640    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/63086243-e82d-4fc5-bfef-e4241c22ad4e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2w8jt\" (UID: \"63086243-e82d-4fc5-bfef-e4241c22ad4e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2w8jt"
Oct 30 13:02:49.816922 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.816704    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5cbda802-f819-4ea6-846e-440cea4afed2-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lt2qg\" (UID: \"5cbda802-f819-4ea6-846e-440cea4afed2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lt2qg"
Oct 30 13:02:49.817716 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.816395    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/feda4324-4a5f-491b-a04c-202c222261a0-ca-trust-extracted\") pod \"image-registry-5b97bd7c47-dssmx\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") " pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:02:49.818284 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.818256    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/feda4324-4a5f-491b-a04c-202c222261a0-image-registry-private-configuration\") pod \"image-registry-5b97bd7c47-dssmx\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") " pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:02:49.818587 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.818564    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-installation-pull-secrets\") pod \"image-registry-9677cf66b-kpdfl\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") " pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:02:49.818587 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:49.818584    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 13:02:49.818734 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:49.818598    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9677cf66b-kpdfl: secret "image-registry-tls" not found
Oct 30 13:02:49.818734 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.818626    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-stats-auth\") pod \"router-default-668b54766c-wgk28\" (UID: \"d5db3ba8-f1de-4fa5-9839-e7e8200da58f\") " pod="openshift-ingress/router-default-668b54766c-wgk28"
Oct 30 13:02:49.818734 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:49.818640    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-registry-tls podName:b4fd1eb6-625a-4649-ba27-9e8798fd1bc5 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:50.318626605 +0000 UTC m=+32.199861835 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-registry-tls") pod "image-registry-9677cf66b-kpdfl" (UID: "b4fd1eb6-625a-4649-ba27-9e8798fd1bc5") : secret "image-registry-tls" not found
Oct 30 13:02:49.821397 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.821375    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-22fm4"
Oct 30 13:02:49.828948 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.828924    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-image-registry-private-configuration\") pod \"image-registry-9677cf66b-kpdfl\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") " pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:02:49.829059 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.828945    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5cbda802-f819-4ea6-846e-440cea4afed2-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lt2qg\" (UID: \"5cbda802-f819-4ea6-846e-440cea4afed2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lt2qg"
Oct 30 13:02:49.829155 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.829086    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/feda4324-4a5f-491b-a04c-202c222261a0-registry-certificates\") pod \"image-registry-5b97bd7c47-dssmx\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") " pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:02:49.829717 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.829694    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-trusted-ca\") pod \"image-registry-9677cf66b-kpdfl\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") " pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:02:49.829873 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.829851    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/feda4324-4a5f-491b-a04c-202c222261a0-trusted-ca\") pod \"image-registry-5b97bd7c47-dssmx\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") " pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:02:49.829954 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.829874    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-ca-trust-extracted\") pod \"image-registry-9677cf66b-kpdfl\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") " pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:02:49.830199 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.830182    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/feda4324-4a5f-491b-a04c-202c222261a0-installation-pull-secrets\") pod \"image-registry-5b97bd7c47-dssmx\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") " pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:02:49.830265 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.830209    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-registry-certificates\") pod \"image-registry-9677cf66b-kpdfl\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") " pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:02:49.830744 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.830729    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-default-certificate\") pod \"router-default-668b54766c-wgk28\" (UID: \"d5db3ba8-f1de-4fa5-9839-e7e8200da58f\") " pod="openshift-ingress/router-default-668b54766c-wgk28"
Oct 30 13:02:49.839938 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.839905    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncqh7\" (UniqueName: \"kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-kube-api-access-ncqh7\") pod \"image-registry-5b97bd7c47-dssmx\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") " pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:02:49.840295 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.840268    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfxrl\" (UniqueName: \"kubernetes.io/projected/abf08e3d-eec3-4f73-92a6-4ffb691cbd2d-kube-api-access-tfxrl\") pod \"volume-data-source-validator-58c9f5f46d-lvf7h\" (UID: \"abf08e3d-eec3-4f73-92a6-4ffb691cbd2d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-lvf7h"
Oct 30 13:02:49.840395 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.840346    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-bound-sa-token\") pod \"image-registry-5b97bd7c47-dssmx\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") " pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:02:49.840452 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.840401    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-bound-sa-token\") pod \"image-registry-9677cf66b-kpdfl\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") " pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:02:49.840633 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.840611    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48dsv\" (UniqueName: \"kubernetes.io/projected/5cbda802-f819-4ea6-846e-440cea4afed2-kube-api-access-48dsv\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lt2qg\" (UID: \"5cbda802-f819-4ea6-846e-440cea4afed2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lt2qg"
Oct 30 13:02:49.840743 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.840725    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdrpm\" (UniqueName: \"kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-kube-api-access-vdrpm\") pod \"image-registry-9677cf66b-kpdfl\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") " pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:02:49.840794 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.840747    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rflhr\" (UniqueName: \"kubernetes.io/projected/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-kube-api-access-rflhr\") pod \"router-default-668b54766c-wgk28\" (UID: \"d5db3ba8-f1de-4fa5-9839-e7e8200da58f\") " pod="openshift-ingress/router-default-668b54766c-wgk28"
Oct 30 13:02:49.851357 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.851338    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-wnzh2"
Oct 30 13:02:49.867921 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.867902    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-lgssx"
Oct 30 13:02:49.917070 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.917045    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/79664154-5454-4294-baad-12bdfad6fe68-config\") pod \"service-ca-operator-544d749c8c-fhnzt\" (UID: \"79664154-5454-4294-baad-12bdfad6fe68\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fhnzt"
Oct 30 13:02:49.917149 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.917086    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f93ee017-ab79-447c-90e0-5b85b71dba24-trusted-ca\") pod \"console-operator-7cd56b8dbc-6jdj4\" (UID: \"f93ee017-ab79-447c-90e0-5b85b71dba24\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6jdj4"
Oct 30 13:02:49.917149 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.917107    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lkrg\" (UniqueName: \"kubernetes.io/projected/f93ee017-ab79-447c-90e0-5b85b71dba24-kube-api-access-2lkrg\") pod \"console-operator-7cd56b8dbc-6jdj4\" (UID: \"f93ee017-ab79-447c-90e0-5b85b71dba24\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6jdj4"
Oct 30 13:02:49.917149 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.917123    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d0b2655-d52d-4c95-9589-d45237fa1057-metrics-tls\") pod \"dns-default-8nb7s\" (UID: \"0d0b2655-d52d-4c95-9589-d45237fa1057\") " pod="openshift-dns/dns-default-8nb7s"
Oct 30 13:02:49.917243 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.917160    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk568\" (UniqueName: \"kubernetes.io/projected/1a25d5a7-83d4-443b-98ae-fc7b603cfc5f-kube-api-access-qk568\") pod \"ingress-canary-wmb4b\" (UID: \"1a25d5a7-83d4-443b-98ae-fc7b603cfc5f\") " pod="openshift-ingress-canary/ingress-canary-wmb4b"
Oct 30 13:02:49.917243 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.917178    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nrlb\" (UniqueName: \"kubernetes.io/projected/79664154-5454-4294-baad-12bdfad6fe68-kube-api-access-7nrlb\") pod \"service-ca-operator-544d749c8c-fhnzt\" (UID: \"79664154-5454-4294-baad-12bdfad6fe68\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fhnzt"
Oct 30 13:02:49.917243 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.917196    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/79664154-5454-4294-baad-12bdfad6fe68-serving-cert\") pod \"service-ca-operator-544d749c8c-fhnzt\" (UID: \"79664154-5454-4294-baad-12bdfad6fe68\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fhnzt"
Oct 30 13:02:49.917243 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.917220    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1a25d5a7-83d4-443b-98ae-fc7b603cfc5f-cert\") pod \"ingress-canary-wmb4b\" (UID: \"1a25d5a7-83d4-443b-98ae-fc7b603cfc5f\") " pod="openshift-ingress-canary/ingress-canary-wmb4b"
Oct 30 13:02:49.917355 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.917255    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e640effb-1ff4-40f2-8d7b-53732edcd2bf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-x8wqt\" (UID: \"e640effb-1ff4-40f2-8d7b-53732edcd2bf\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x8wqt"
Oct 30 13:02:49.917355 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.917296    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e640effb-1ff4-40f2-8d7b-53732edcd2bf-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-x8wqt\" (UID: \"e640effb-1ff4-40f2-8d7b-53732edcd2bf\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x8wqt"
Oct 30 13:02:49.917355 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.917312    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/63086243-e82d-4fc5-bfef-e4241c22ad4e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2w8jt\" (UID: \"63086243-e82d-4fc5-bfef-e4241c22ad4e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2w8jt"
Oct 30 13:02:49.917355 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.917333    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d0b2655-d52d-4c95-9589-d45237fa1057-tmp-dir\") pod \"dns-default-8nb7s\" (UID: \"0d0b2655-d52d-4c95-9589-d45237fa1057\") " pod="openshift-dns/dns-default-8nb7s"
Oct 30 13:02:49.917355 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.917350    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r598w\" (UniqueName: \"kubernetes.io/projected/e640effb-1ff4-40f2-8d7b-53732edcd2bf-kube-api-access-r598w\") pod \"cluster-monitoring-operator-884576c57-x8wqt\" (UID: \"e640effb-1ff4-40f2-8d7b-53732edcd2bf\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x8wqt"
Oct 30 13:02:49.917490 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.917378    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f93ee017-ab79-447c-90e0-5b85b71dba24-config\") pod \"console-operator-7cd56b8dbc-6jdj4\" (UID: \"f93ee017-ab79-447c-90e0-5b85b71dba24\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6jdj4"
Oct 30 13:02:49.917490 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.917414    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f93ee017-ab79-447c-90e0-5b85b71dba24-serving-cert\") pod \"console-operator-7cd56b8dbc-6jdj4\" (UID: \"f93ee017-ab79-447c-90e0-5b85b71dba24\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6jdj4"
Oct 30 13:02:49.917490 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.917428    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnx68\" (UniqueName: \"kubernetes.io/projected/4ad66a0c-8dc0-40a4-a1b2-28d26b0de94b-kube-api-access-rnx68\") pod \"network-check-source-6f78d59f99-r7vrw\" (UID: \"4ad66a0c-8dc0-40a4-a1b2-28d26b0de94b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-r7vrw"
Oct 30 13:02:49.917490 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.917457    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plb6z\" (UniqueName: \"kubernetes.io/projected/0d0b2655-d52d-4c95-9589-d45237fa1057-kube-api-access-plb6z\") pod \"dns-default-8nb7s\" (UID: \"0d0b2655-d52d-4c95-9589-d45237fa1057\") " pod="openshift-dns/dns-default-8nb7s"
Oct 30 13:02:49.917490 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.917478    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/63086243-e82d-4fc5-bfef-e4241c22ad4e-nginx-conf\") pod \"networking-console-plugin-86c478bb88-2w8jt\" (UID: \"63086243-e82d-4fc5-bfef-e4241c22ad4e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2w8jt"
Oct 30 13:02:49.917632 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.917500    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0d0b2655-d52d-4c95-9589-d45237fa1057-config-volume\") pod \"dns-default-8nb7s\" (UID: \"0d0b2655-d52d-4c95-9589-d45237fa1057\") " pod="openshift-dns/dns-default-8nb7s"
Oct 30 13:02:49.918044 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.918026    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0d0b2655-d52d-4c95-9589-d45237fa1057-config-volume\") pod \"dns-default-8nb7s\" (UID: \"0d0b2655-d52d-4c95-9589-d45237fa1057\") " pod="openshift-dns/dns-default-8nb7s"
Oct 30 13:02:49.918201 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.918179    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/79664154-5454-4294-baad-12bdfad6fe68-config\") pod \"service-ca-operator-544d749c8c-fhnzt\" (UID: \"79664154-5454-4294-baad-12bdfad6fe68\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fhnzt"
Oct 30 13:02:49.918970 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.918938    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f93ee017-ab79-447c-90e0-5b85b71dba24-trusted-ca\") pod \"console-operator-7cd56b8dbc-6jdj4\" (UID: \"f93ee017-ab79-447c-90e0-5b85b71dba24\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6jdj4"
Oct 30 13:02:49.919844 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.919099    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f93ee017-ab79-447c-90e0-5b85b71dba24-config\") pod \"console-operator-7cd56b8dbc-6jdj4\" (UID: \"f93ee017-ab79-447c-90e0-5b85b71dba24\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6jdj4"
Oct 30 13:02:49.919844 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:49.919181    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 13:02:49.919844 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:49.919229    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d0b2655-d52d-4c95-9589-d45237fa1057-metrics-tls podName:0d0b2655-d52d-4c95-9589-d45237fa1057 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:50.419214062 +0000 UTC m=+32.300449298 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0d0b2655-d52d-4c95-9589-d45237fa1057-metrics-tls") pod "dns-default-8nb7s" (UID: "0d0b2655-d52d-4c95-9589-d45237fa1057") : secret "dns-default-metrics-tls" not found
Oct 30 13:02:49.919844 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:49.919437    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 13:02:49.919844 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:49.919499    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e640effb-1ff4-40f2-8d7b-53732edcd2bf-cluster-monitoring-operator-tls podName:e640effb-1ff4-40f2-8d7b-53732edcd2bf nodeName:}" failed. No retries permitted until 2026-10-30 13:02:50.419482096 +0000 UTC m=+32.300717317 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e640effb-1ff4-40f2-8d7b-53732edcd2bf-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-x8wqt" (UID: "e640effb-1ff4-40f2-8d7b-53732edcd2bf") : secret "cluster-monitoring-operator-tls" not found
Oct 30 13:02:49.919844 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:49.919500    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 13:02:49.919844 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:49.919541    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63086243-e82d-4fc5-bfef-e4241c22ad4e-networking-console-plugin-cert podName:63086243-e82d-4fc5-bfef-e4241c22ad4e nodeName:}" failed. No retries permitted until 2026-10-30 13:02:50.419531543 +0000 UTC m=+32.300766771 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/63086243-e82d-4fc5-bfef-e4241c22ad4e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2w8jt" (UID: "63086243-e82d-4fc5-bfef-e4241c22ad4e") : secret "networking-console-plugin-cert" not found
Oct 30 13:02:49.919844 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.919803    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/63086243-e82d-4fc5-bfef-e4241c22ad4e-nginx-conf\") pod \"networking-console-plugin-86c478bb88-2w8jt\" (UID: \"63086243-e82d-4fc5-bfef-e4241c22ad4e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2w8jt"
Oct 30 13:02:49.919844 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.919824    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d0b2655-d52d-4c95-9589-d45237fa1057-tmp-dir\") pod \"dns-default-8nb7s\" (UID: \"0d0b2655-d52d-4c95-9589-d45237fa1057\") " pod="openshift-dns/dns-default-8nb7s"
Oct 30 13:02:49.920190 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.919974    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e640effb-1ff4-40f2-8d7b-53732edcd2bf-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-x8wqt\" (UID: \"e640effb-1ff4-40f2-8d7b-53732edcd2bf\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x8wqt"
Oct 30 13:02:49.921367 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.921346    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1a25d5a7-83d4-443b-98ae-fc7b603cfc5f-cert\") pod \"ingress-canary-wmb4b\" (UID: \"1a25d5a7-83d4-443b-98ae-fc7b603cfc5f\") " pod="openshift-ingress-canary/ingress-canary-wmb4b"
Oct 30 13:02:49.922482 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.922461    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/79664154-5454-4294-baad-12bdfad6fe68-serving-cert\") pod \"service-ca-operator-544d749c8c-fhnzt\" (UID: \"79664154-5454-4294-baad-12bdfad6fe68\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fhnzt"
Oct 30 13:02:49.924191 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.924169    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f93ee017-ab79-447c-90e0-5b85b71dba24-serving-cert\") pod \"console-operator-7cd56b8dbc-6jdj4\" (UID: \"f93ee017-ab79-447c-90e0-5b85b71dba24\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6jdj4"
Oct 30 13:02:49.939237 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.939099    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nrlb\" (UniqueName: \"kubernetes.io/projected/79664154-5454-4294-baad-12bdfad6fe68-kube-api-access-7nrlb\") pod \"service-ca-operator-544d749c8c-fhnzt\" (UID: \"79664154-5454-4294-baad-12bdfad6fe68\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fhnzt"
Oct 30 13:02:49.939237 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.939152    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plb6z\" (UniqueName: \"kubernetes.io/projected/0d0b2655-d52d-4c95-9589-d45237fa1057-kube-api-access-plb6z\") pod \"dns-default-8nb7s\" (UID: \"0d0b2655-d52d-4c95-9589-d45237fa1057\") " pod="openshift-dns/dns-default-8nb7s"
Oct 30 13:02:49.940934 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.940052    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk568\" (UniqueName: \"kubernetes.io/projected/1a25d5a7-83d4-443b-98ae-fc7b603cfc5f-kube-api-access-qk568\") pod \"ingress-canary-wmb4b\" (UID: \"1a25d5a7-83d4-443b-98ae-fc7b603cfc5f\") " pod="openshift-ingress-canary/ingress-canary-wmb4b"
Oct 30 13:02:49.940934 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.940632    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnx68\" (UniqueName: \"kubernetes.io/projected/4ad66a0c-8dc0-40a4-a1b2-28d26b0de94b-kube-api-access-rnx68\") pod \"network-check-source-6f78d59f99-r7vrw\" (UID: \"4ad66a0c-8dc0-40a4-a1b2-28d26b0de94b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-r7vrw"
Oct 30 13:02:49.940934 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.940691    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lkrg\" (UniqueName: \"kubernetes.io/projected/f93ee017-ab79-447c-90e0-5b85b71dba24-kube-api-access-2lkrg\") pod \"console-operator-7cd56b8dbc-6jdj4\" (UID: \"f93ee017-ab79-447c-90e0-5b85b71dba24\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6jdj4"
Oct 30 13:02:49.940934 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.940756    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wmb4b"
Oct 30 13:02:49.942579 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.942539    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r598w\" (UniqueName: \"kubernetes.io/projected/e640effb-1ff4-40f2-8d7b-53732edcd2bf-kube-api-access-r598w\") pod \"cluster-monitoring-operator-884576c57-x8wqt\" (UID: \"e640effb-1ff4-40f2-8d7b-53732edcd2bf\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x8wqt"
Oct 30 13:02:49.947752 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.947697    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lt2qg"
Oct 30 13:02:49.997034 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:49.996995    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-lvf7h"
Oct 30 13:02:50.084674 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:50.084441    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-6jdj4"
Oct 30 13:02:50.129546 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:50.129521    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fhnzt"
Oct 30 13:02:50.182246 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:50.182210    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-r7vrw"
Oct 30 13:02:50.204338 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:50.204301    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-22fm4"]
Oct 30 13:02:50.206384 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:50.206354    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-rbf7t"]
Oct 30 13:02:50.208412 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:50.208387    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wnzh2"]
Oct 30 13:02:50.220893 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:50.220866    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7cca8a39-bee0-4705-bcde-b31060e44944-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gxcrm\" (UID: \"7cca8a39-bee0-4705-bcde-b31060e44944\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gxcrm"
Oct 30 13:02:50.220893 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:50.220878    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46bedd0c_7237_4128_be98_69e4474bb535.slice/crio-f816757e82dd09f5cfa0a0f30bfaf6c873fadd481a17dfde9e290f9900041ffb WatchSource:0}: Error finding container f816757e82dd09f5cfa0a0f30bfaf6c873fadd481a17dfde9e290f9900041ffb: Status 404 returned error can't find the container with id f816757e82dd09f5cfa0a0f30bfaf6c873fadd481a17dfde9e290f9900041ffb
Oct 30 13:02:50.221096 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:50.221083    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 13:02:50.221176 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:50.221164    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cca8a39-bee0-4705-bcde-b31060e44944-samples-operator-tls podName:7cca8a39-bee0-4705-bcde-b31060e44944 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:51.221144373 +0000 UTC m=+33.102379588 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7cca8a39-bee0-4705-bcde-b31060e44944-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-gxcrm" (UID: "7cca8a39-bee0-4705-bcde-b31060e44944") : secret "samples-operator-tls" not found
Oct 30 13:02:50.221945 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:50.221918    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8021bc0b_4447_4f04_b9bd_5f80e9cf3507.slice/crio-562b94e335ef4ee49a4a2c8c6eea2920298d30c5531be03ac529afce7dc418e4 WatchSource:0}: Error finding container 562b94e335ef4ee49a4a2c8c6eea2920298d30c5531be03ac529afce7dc418e4: Status 404 returned error can't find the container with id 562b94e335ef4ee49a4a2c8c6eea2920298d30c5531be03ac529afce7dc418e4
Oct 30 13:02:50.222755 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:50.222731    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44f513ab_9115_4761_aa8b_c546212ec17b.slice/crio-de331273dc14e36948f28a293bd31355eecd2f4777b203d2a0420a0f26b75b5f WatchSource:0}: Error finding container de331273dc14e36948f28a293bd31355eecd2f4777b203d2a0420a0f26b75b5f: Status 404 returned error can't find the container with id de331273dc14e36948f28a293bd31355eecd2f4777b203d2a0420a0f26b75b5f
Oct 30 13:02:50.237442 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:50.237416    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-lgssx"]
Oct 30 13:02:50.244917 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:50.242916    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod155bbdd4_b566_467f_88c6_ec5b4b881a09.slice/crio-3f90007edff3cc0f6929888fcff5cdf01af5714c69ea75da6d164524fa6c4ba6 WatchSource:0}: Error finding container 3f90007edff3cc0f6929888fcff5cdf01af5714c69ea75da6d164524fa6c4ba6: Status 404 returned error can't find the container with id 3f90007edff3cc0f6929888fcff5cdf01af5714c69ea75da6d164524fa6c4ba6
Oct 30 13:02:50.295930 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:50.295810    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lt2qg"]
Oct 30 13:02:50.297002 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:50.296964    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cbda802_f819_4ea6_846e_440cea4afed2.slice/crio-7569409b6e188b60e656069548c94067881169acb401a01f609ce63fa445e43c WatchSource:0}: Error finding container 7569409b6e188b60e656069548c94067881169acb401a01f609ce63fa445e43c: Status 404 returned error can't find the container with id 7569409b6e188b60e656069548c94067881169acb401a01f609ce63fa445e43c
Oct 30 13:02:50.297144 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:50.297105    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wmb4b"]
Oct 30 13:02:50.299364 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:50.299335    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a25d5a7_83d4_443b_98ae_fc7b603cfc5f.slice/crio-d4c6b7cb6278106280fbe5bc2fb62290e5932aa7d28d851f97901a9af05cdb96 WatchSource:0}: Error finding container d4c6b7cb6278106280fbe5bc2fb62290e5932aa7d28d851f97901a9af05cdb96: Status 404 returned error can't find the container with id d4c6b7cb6278106280fbe5bc2fb62290e5932aa7d28d851f97901a9af05cdb96
Oct 30 13:02:50.319944 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:50.319870    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-lvf7h"]
Oct 30 13:02:50.323854 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:50.322830    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-service-ca-bundle\") pod \"router-default-668b54766c-wgk28\" (UID: \"d5db3ba8-f1de-4fa5-9839-e7e8200da58f\") " pod="openshift-ingress/router-default-668b54766c-wgk28"
Oct 30 13:02:50.323854 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:50.322874    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-registry-tls\") pod \"image-registry-5b97bd7c47-dssmx\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") " pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:02:50.323854 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:50.322903    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-metrics-certs\") pod \"router-default-668b54766c-wgk28\" (UID: \"d5db3ba8-f1de-4fa5-9839-e7e8200da58f\") " pod="openshift-ingress/router-default-668b54766c-wgk28"
Oct 30 13:02:50.323854 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:50.322963    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-registry-tls\") pod \"image-registry-9677cf66b-kpdfl\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") " pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:02:50.323854 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:50.323192    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 13:02:50.323854 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:50.323208    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9677cf66b-kpdfl: secret "image-registry-tls" not found
Oct 30 13:02:50.323854 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:50.323264    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-registry-tls podName:b4fd1eb6-625a-4649-ba27-9e8798fd1bc5 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:51.323245906 +0000 UTC m=+33.204481129 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-registry-tls") pod "image-registry-9677cf66b-kpdfl" (UID: "b4fd1eb6-625a-4649-ba27-9e8798fd1bc5") : secret "image-registry-tls" not found
Oct 30 13:02:50.323854 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:50.323607    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-service-ca-bundle podName:d5db3ba8-f1de-4fa5-9839-e7e8200da58f nodeName:}" failed. No retries permitted until 2026-10-30 13:02:51.323590303 +0000 UTC m=+33.204825503 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-service-ca-bundle") pod "router-default-668b54766c-wgk28" (UID: "d5db3ba8-f1de-4fa5-9839-e7e8200da58f") : configmap references non-existent config key: service-ca.crt
Oct 30 13:02:50.323854 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:50.323692    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 13:02:50.323854 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:50.323703    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b97bd7c47-dssmx: secret "image-registry-tls" not found
Oct 30 13:02:50.323854 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:50.323739    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-registry-tls podName:feda4324-4a5f-491b-a04c-202c222261a0 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:51.323727507 +0000 UTC m=+33.204962708 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-registry-tls") pod "image-registry-5b97bd7c47-dssmx" (UID: "feda4324-4a5f-491b-a04c-202c222261a0") : secret "image-registry-tls" not found
Oct 30 13:02:50.323854 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:50.323794    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 13:02:50.323854 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:50.323823    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-metrics-certs podName:d5db3ba8-f1de-4fa5-9839-e7e8200da58f nodeName:}" failed. No retries permitted until 2026-10-30 13:02:51.323813807 +0000 UTC m=+33.205049007 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-metrics-certs") pod "router-default-668b54766c-wgk28" (UID: "d5db3ba8-f1de-4fa5-9839-e7e8200da58f") : secret "router-metrics-certs-default" not found
Oct 30 13:02:50.330775 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:50.330749    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabf08e3d_eec3_4f73_92a6_4ffb691cbd2d.slice/crio-f6950ee20e87263fb0dd7718cc81691d24936acf6bb32219f5659b8b23cfefdd WatchSource:0}: Error finding container f6950ee20e87263fb0dd7718cc81691d24936acf6bb32219f5659b8b23cfefdd: Status 404 returned error can't find the container with id f6950ee20e87263fb0dd7718cc81691d24936acf6bb32219f5659b8b23cfefdd
Oct 30 13:02:50.398392 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:50.398328    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-6jdj4"]
Oct 30 13:02:50.402153 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:50.402116    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf93ee017_ab79_447c_90e0_5b85b71dba24.slice/crio-46ff484b065035d2962579b1722fe86d2b4e9d2e4b5cea97bf770ae859b773bd WatchSource:0}: Error finding container 46ff484b065035d2962579b1722fe86d2b4e9d2e4b5cea97bf770ae859b773bd: Status 404 returned error can't find the container with id 46ff484b065035d2962579b1722fe86d2b4e9d2e4b5cea97bf770ae859b773bd
Oct 30 13:02:50.424437 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:50.424413    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d0b2655-d52d-4c95-9589-d45237fa1057-metrics-tls\") pod \"dns-default-8nb7s\" (UID: \"0d0b2655-d52d-4c95-9589-d45237fa1057\") " pod="openshift-dns/dns-default-8nb7s"
Oct 30 13:02:50.424523 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:50.424469    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e640effb-1ff4-40f2-8d7b-53732edcd2bf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-x8wqt\" (UID: \"e640effb-1ff4-40f2-8d7b-53732edcd2bf\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x8wqt"
Oct 30 13:02:50.424601 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:50.424577    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 13:02:50.424657 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:50.424643    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 13:02:50.424704 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:50.424584    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/63086243-e82d-4fc5-bfef-e4241c22ad4e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2w8jt\" (UID: \"63086243-e82d-4fc5-bfef-e4241c22ad4e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2w8jt"
Oct 30 13:02:50.424704 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:50.424664    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d0b2655-d52d-4c95-9589-d45237fa1057-metrics-tls podName:0d0b2655-d52d-4c95-9589-d45237fa1057 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:51.424644867 +0000 UTC m=+33.305880066 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0d0b2655-d52d-4c95-9589-d45237fa1057-metrics-tls") pod "dns-default-8nb7s" (UID: "0d0b2655-d52d-4c95-9589-d45237fa1057") : secret "dns-default-metrics-tls" not found
Oct 30 13:02:50.424704 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:50.424581    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 13:02:50.424704 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:50.424700    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63086243-e82d-4fc5-bfef-e4241c22ad4e-networking-console-plugin-cert podName:63086243-e82d-4fc5-bfef-e4241c22ad4e nodeName:}" failed. No retries permitted until 2026-10-30 13:02:51.424683428 +0000 UTC m=+33.305918626 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/63086243-e82d-4fc5-bfef-e4241c22ad4e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2w8jt" (UID: "63086243-e82d-4fc5-bfef-e4241c22ad4e") : secret "networking-console-plugin-cert" not found
Oct 30 13:02:50.424877 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:50.424721    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e640effb-1ff4-40f2-8d7b-53732edcd2bf-cluster-monitoring-operator-tls podName:e640effb-1ff4-40f2-8d7b-53732edcd2bf nodeName:}" failed. No retries permitted until 2026-10-30 13:02:51.424712687 +0000 UTC m=+33.305947887 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e640effb-1ff4-40f2-8d7b-53732edcd2bf-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-x8wqt" (UID: "e640effb-1ff4-40f2-8d7b-53732edcd2bf") : secret "cluster-monitoring-operator-tls" not found
Oct 30 13:02:50.437878 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:50.437822    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-fhnzt"]
Oct 30 13:02:50.442111 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:50.442089    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79664154_5454_4294_baad_12bdfad6fe68.slice/crio-12715619dec36263a869e0698ef7e31b9481effac5627057399505c660155ba4 WatchSource:0}: Error finding container 12715619dec36263a869e0698ef7e31b9481effac5627057399505c660155ba4: Status 404 returned error can't find the container with id 12715619dec36263a869e0698ef7e31b9481effac5627057399505c660155ba4
Oct 30 13:02:50.514601 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:50.514569    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-r7vrw"]
Oct 30 13:02:50.519367 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:50.519338    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ad66a0c_8dc0_40a4_a1b2_28d26b0de94b.slice/crio-0aef0d1cf6af23a8dab50f6eb1d692bf62f8538e1dc9f793a6d8680e303e4732 WatchSource:0}: Error finding container 0aef0d1cf6af23a8dab50f6eb1d692bf62f8538e1dc9f793a6d8680e303e4732: Status 404 returned error can't find the container with id 0aef0d1cf6af23a8dab50f6eb1d692bf62f8538e1dc9f793a6d8680e303e4732
Oct 30 13:02:50.975952 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:50.975916    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-lvf7h" event={"ID":"abf08e3d-eec3-4f73-92a6-4ffb691cbd2d","Type":"ContainerStarted","Data":"f6950ee20e87263fb0dd7718cc81691d24936acf6bb32219f5659b8b23cfefdd"}
Oct 30 13:02:50.978561 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:50.978522    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wmb4b" event={"ID":"1a25d5a7-83d4-443b-98ae-fc7b603cfc5f","Type":"ContainerStarted","Data":"d4c6b7cb6278106280fbe5bc2fb62290e5932aa7d28d851f97901a9af05cdb96"}
Oct 30 13:02:50.980705 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:50.980624    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-wnzh2" event={"ID":"44f513ab-9115-4761-aa8b-c546212ec17b","Type":"ContainerStarted","Data":"497030d9f6e5f93000676085636157f21a890bc7b3e5a86850034ad62832b09a"}
Oct 30 13:02:50.980705 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:50.980670    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-wnzh2" event={"ID":"44f513ab-9115-4761-aa8b-c546212ec17b","Type":"ContainerStarted","Data":"de331273dc14e36948f28a293bd31355eecd2f4777b203d2a0420a0f26b75b5f"}
Oct 30 13:02:50.983163 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:50.982653    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-22fm4" event={"ID":"46bedd0c-7237-4128-be98-69e4474bb535","Type":"ContainerStarted","Data":"faa225ca1716821ce9e581459404fa5260c10f5c064ed72fb8038e1f487af52f"}
Oct 30 13:02:50.983163 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:50.982694    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-22fm4" event={"ID":"46bedd0c-7237-4128-be98-69e4474bb535","Type":"ContainerStarted","Data":"f816757e82dd09f5cfa0a0f30bfaf6c873fadd481a17dfde9e290f9900041ffb"}
Oct 30 13:02:50.985112 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:50.985084    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-rbf7t" event={"ID":"8021bc0b-4447-4f04-b9bd-5f80e9cf3507","Type":"ContainerStarted","Data":"2408e7312889d739124476c3fe3e56f27558b047ff14e84a0426137ad59b4c79"}
Oct 30 13:02:50.985229 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:50.985145    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-rbf7t" event={"ID":"8021bc0b-4447-4f04-b9bd-5f80e9cf3507","Type":"ContainerStarted","Data":"562b94e335ef4ee49a4a2c8c6eea2920298d30c5531be03ac529afce7dc418e4"}
Oct 30 13:02:50.987354 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:50.987312    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-r7vrw" event={"ID":"4ad66a0c-8dc0-40a4-a1b2-28d26b0de94b","Type":"ContainerStarted","Data":"0aef0d1cf6af23a8dab50f6eb1d692bf62f8538e1dc9f793a6d8680e303e4732"}
Oct 30 13:02:50.991037 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:50.991010    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fhnzt" event={"ID":"79664154-5454-4294-baad-12bdfad6fe68","Type":"ContainerStarted","Data":"12715619dec36263a869e0698ef7e31b9481effac5627057399505c660155ba4"}
Oct 30 13:02:50.994077 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:50.994051    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-6jdj4" event={"ID":"f93ee017-ab79-447c-90e0-5b85b71dba24","Type":"ContainerStarted","Data":"46ff484b065035d2962579b1722fe86d2b4e9d2e4b5cea97bf770ae859b773bd"}
Oct 30 13:02:51.002867 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:51.002719    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lt2qg" event={"ID":"5cbda802-f819-4ea6-846e-440cea4afed2","Type":"ContainerStarted","Data":"7569409b6e188b60e656069548c94067881169acb401a01f609ce63fa445e43c"}
Oct 30 13:02:51.006319 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:51.006281    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-lgssx" event={"ID":"155bbdd4-b566-467f-88c6-ec5b4b881a09","Type":"ContainerStarted","Data":"3f90007edff3cc0f6929888fcff5cdf01af5714c69ea75da6d164524fa6c4ba6"}
Oct 30 13:02:51.059229 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:51.059173    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-wnzh2" podStartSLOduration=400.059158534 podStartE2EDuration="6m40.059158534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:56:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:51.019811161 +0000 UTC m=+32.901046447" watchObservedRunningTime="2026-10-30 13:02:51.059158534 +0000 UTC m=+32.940393754"
Oct 30 13:02:51.059994 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:51.059948    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-rbf7t" podStartSLOduration=400.059938556 podStartE2EDuration="6m40.059938556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:56:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:51.057928879 +0000 UTC m=+32.939164100" watchObservedRunningTime="2026-10-30 13:02:51.059938556 +0000 UTC m=+32.941173777"
Oct 30 13:02:51.234614 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:51.234516    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7cca8a39-bee0-4705-bcde-b31060e44944-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gxcrm\" (UID: \"7cca8a39-bee0-4705-bcde-b31060e44944\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gxcrm"
Oct 30 13:02:51.235518 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:51.234930    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 13:02:51.235518 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:51.235004    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cca8a39-bee0-4705-bcde-b31060e44944-samples-operator-tls podName:7cca8a39-bee0-4705-bcde-b31060e44944 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:53.234986241 +0000 UTC m=+35.116221456 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7cca8a39-bee0-4705-bcde-b31060e44944-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-gxcrm" (UID: "7cca8a39-bee0-4705-bcde-b31060e44944") : secret "samples-operator-tls" not found
Oct 30 13:02:51.336636 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:51.335479    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-service-ca-bundle\") pod \"router-default-668b54766c-wgk28\" (UID: \"d5db3ba8-f1de-4fa5-9839-e7e8200da58f\") " pod="openshift-ingress/router-default-668b54766c-wgk28"
Oct 30 13:02:51.336636 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:51.335534    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-registry-tls\") pod \"image-registry-5b97bd7c47-dssmx\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") " pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:02:51.336636 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:51.335564    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-metrics-certs\") pod \"router-default-668b54766c-wgk28\" (UID: \"d5db3ba8-f1de-4fa5-9839-e7e8200da58f\") " pod="openshift-ingress/router-default-668b54766c-wgk28"
Oct 30 13:02:51.336636 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:51.335626    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-registry-tls\") pod \"image-registry-9677cf66b-kpdfl\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") " pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:02:51.336636 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:51.335823    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 13:02:51.336636 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:51.335838    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9677cf66b-kpdfl: secret "image-registry-tls" not found
Oct 30 13:02:51.336636 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:51.335897    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-registry-tls podName:b4fd1eb6-625a-4649-ba27-9e8798fd1bc5 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:53.335879573 +0000 UTC m=+35.217114790 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-registry-tls") pod "image-registry-9677cf66b-kpdfl" (UID: "b4fd1eb6-625a-4649-ba27-9e8798fd1bc5") : secret "image-registry-tls" not found
Oct 30 13:02:51.336636 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:51.336361    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-service-ca-bundle podName:d5db3ba8-f1de-4fa5-9839-e7e8200da58f nodeName:}" failed. No retries permitted until 2026-10-30 13:02:53.336340333 +0000 UTC m=+35.217575549 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-service-ca-bundle") pod "router-default-668b54766c-wgk28" (UID: "d5db3ba8-f1de-4fa5-9839-e7e8200da58f") : configmap references non-existent config key: service-ca.crt
Oct 30 13:02:51.336636 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:51.336444    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 13:02:51.336636 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:51.336455    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b97bd7c47-dssmx: secret "image-registry-tls" not found
Oct 30 13:02:51.336636 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:51.336488    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-registry-tls podName:feda4324-4a5f-491b-a04c-202c222261a0 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:53.336477264 +0000 UTC m=+35.217712476 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-registry-tls") pod "image-registry-5b97bd7c47-dssmx" (UID: "feda4324-4a5f-491b-a04c-202c222261a0") : secret "image-registry-tls" not found
Oct 30 13:02:51.336636 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:51.336551    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 13:02:51.336636 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:51.336581    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-metrics-certs podName:d5db3ba8-f1de-4fa5-9839-e7e8200da58f nodeName:}" failed. No retries permitted until 2026-10-30 13:02:53.336571221 +0000 UTC m=+35.217806433 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-metrics-certs") pod "router-default-668b54766c-wgk28" (UID: "d5db3ba8-f1de-4fa5-9839-e7e8200da58f") : secret "router-metrics-certs-default" not found
Oct 30 13:02:51.438424 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:51.436756    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e640effb-1ff4-40f2-8d7b-53732edcd2bf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-x8wqt\" (UID: \"e640effb-1ff4-40f2-8d7b-53732edcd2bf\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x8wqt"
Oct 30 13:02:51.438424 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:51.436812    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a32a253a-ecaf-4f90-af1c-27a7e53b9629-metrics-certs\") pod \"network-metrics-daemon-mbncl\" (UID: \"a32a253a-ecaf-4f90-af1c-27a7e53b9629\") " pod="openshift-multus/network-metrics-daemon-mbncl"
Oct 30 13:02:51.438424 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:51.436845    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/10b774a1-1476-4f91-b6e5-d52b3f45700e-original-pull-secret\") pod \"global-pull-secret-syncer-wrfqz\" (UID: \"10b774a1-1476-4f91-b6e5-d52b3f45700e\") " pod="kube-system/global-pull-secret-syncer-wrfqz"
Oct 30 13:02:51.438424 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:51.436880    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/63086243-e82d-4fc5-bfef-e4241c22ad4e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2w8jt\" (UID: \"63086243-e82d-4fc5-bfef-e4241c22ad4e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2w8jt"
Oct 30 13:02:51.438424 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:51.437011    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d0b2655-d52d-4c95-9589-d45237fa1057-metrics-tls\") pod \"dns-default-8nb7s\" (UID: \"0d0b2655-d52d-4c95-9589-d45237fa1057\") " pod="openshift-dns/dns-default-8nb7s"
Oct 30 13:02:51.438424 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:51.437172    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 13:02:51.438424 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:51.437246    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d0b2655-d52d-4c95-9589-d45237fa1057-metrics-tls podName:0d0b2655-d52d-4c95-9589-d45237fa1057 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:53.437225764 +0000 UTC m=+35.318460976 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0d0b2655-d52d-4c95-9589-d45237fa1057-metrics-tls") pod "dns-default-8nb7s" (UID: "0d0b2655-d52d-4c95-9589-d45237fa1057") : secret "dns-default-metrics-tls" not found
Oct 30 13:02:51.438424 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:51.437652    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 13:02:51.438424 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:51.437701    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e640effb-1ff4-40f2-8d7b-53732edcd2bf-cluster-monitoring-operator-tls podName:e640effb-1ff4-40f2-8d7b-53732edcd2bf nodeName:}" failed. No retries permitted until 2026-10-30 13:02:53.437687081 +0000 UTC m=+35.318922293 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e640effb-1ff4-40f2-8d7b-53732edcd2bf-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-x8wqt" (UID: "e640effb-1ff4-40f2-8d7b-53732edcd2bf") : secret "cluster-monitoring-operator-tls" not found
Oct 30 13:02:51.438424 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:51.437757    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 13:02:51.438424 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:51.437789    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a32a253a-ecaf-4f90-af1c-27a7e53b9629-metrics-certs podName:a32a253a-ecaf-4f90-af1c-27a7e53b9629 nodeName:}" failed. No retries permitted until 2026-10-30 13:03:23.437778647 +0000 UTC m=+65.319013859 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a32a253a-ecaf-4f90-af1c-27a7e53b9629-metrics-certs") pod "network-metrics-daemon-mbncl" (UID: "a32a253a-ecaf-4f90-af1c-27a7e53b9629") : secret "metrics-daemon-secret" not found
Oct 30 13:02:51.438424 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:51.438315    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 13:02:51.438424 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:51.438365    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63086243-e82d-4fc5-bfef-e4241c22ad4e-networking-console-plugin-cert podName:63086243-e82d-4fc5-bfef-e4241c22ad4e nodeName:}" failed. No retries permitted until 2026-10-30 13:02:53.438351724 +0000 UTC m=+35.319586935 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/63086243-e82d-4fc5-bfef-e4241c22ad4e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2w8jt" (UID: "63086243-e82d-4fc5-bfef-e4241c22ad4e") : secret "networking-console-plugin-cert" not found
Oct 30 13:02:51.441408 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:51.441356    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/10b774a1-1476-4f91-b6e5-d52b3f45700e-original-pull-secret\") pod \"global-pull-secret-syncer-wrfqz\" (UID: \"10b774a1-1476-4f91-b6e5-d52b3f45700e\") " pod="kube-system/global-pull-secret-syncer-wrfqz"
Oct 30 13:02:51.539299 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:51.538608    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz5ls\" (UniqueName: \"kubernetes.io/projected/a07f5a09-920e-462f-ab6d-7a2ac53d199e-kube-api-access-gz5ls\") pod \"network-check-target-rxd99\" (UID: \"a07f5a09-920e-462f-ab6d-7a2ac53d199e\") " pod="openshift-network-diagnostics/network-check-target-rxd99"
Oct 30 13:02:51.555142 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:51.555064    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz5ls\" (UniqueName: \"kubernetes.io/projected/a07f5a09-920e-462f-ab6d-7a2ac53d199e-kube-api-access-gz5ls\") pod \"network-check-target-rxd99\" (UID: \"a07f5a09-920e-462f-ab6d-7a2ac53d199e\") " pod="openshift-network-diagnostics/network-check-target-rxd99"
Oct 30 13:02:51.561866 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:51.561623    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrfqz"
Oct 30 13:02:51.812001 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:51.811926    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxd99"
Oct 30 13:02:51.888168 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:51.887324    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-22fm4" podStartSLOduration=400.887302548 podStartE2EDuration="6m40.887302548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:56:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:51.094659476 +0000 UTC m=+32.975894697" watchObservedRunningTime="2026-10-30 13:02:51.887302548 +0000 UTC m=+33.768537771"
Oct 30 13:02:51.888707 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:51.888631    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wrfqz"]
Oct 30 13:02:51.915931 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:51.915885    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10b774a1_1476_4f91_b6e5_d52b3f45700e.slice/crio-0b84f92815296a8a34a2754e0fa1c4510032d61e7fd9551ebf454424a49a192f WatchSource:0}: Error finding container 0b84f92815296a8a34a2754e0fa1c4510032d61e7fd9551ebf454424a49a192f: Status 404 returned error can't find the container with id 0b84f92815296a8a34a2754e0fa1c4510032d61e7fd9551ebf454424a49a192f
Oct 30 13:02:52.015768 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:52.015683    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wrfqz" event={"ID":"10b774a1-1476-4f91-b6e5-d52b3f45700e","Type":"ContainerStarted","Data":"0b84f92815296a8a34a2754e0fa1c4510032d61e7fd9551ebf454424a49a192f"}
Oct 30 13:02:52.159141 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:52.158991    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rxd99"]
Oct 30 13:02:52.163172 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:52.163140    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda07f5a09_920e_462f_ab6d_7a2ac53d199e.slice/crio-1b27ed0977f245135de6ee6d9d6d381ba53831c5ba22d2699bab4c5db56f508c WatchSource:0}: Error finding container 1b27ed0977f245135de6ee6d9d6d381ba53831c5ba22d2699bab4c5db56f508c: Status 404 returned error can't find the container with id 1b27ed0977f245135de6ee6d9d6d381ba53831c5ba22d2699bab4c5db56f508c
Oct 30 13:02:53.021218 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:53.021175    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-lvf7h" event={"ID":"abf08e3d-eec3-4f73-92a6-4ffb691cbd2d","Type":"ContainerStarted","Data":"56eb27bcb1e52aaff938588f5cdfdd7d3cd9a863ff11dc8078ee3d2b39e83811"}
Oct 30 13:02:53.026669 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:53.026631    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rxd99" event={"ID":"a07f5a09-920e-462f-ab6d-7a2ac53d199e","Type":"ContainerStarted","Data":"1b27ed0977f245135de6ee6d9d6d381ba53831c5ba22d2699bab4c5db56f508c"}
Oct 30 13:02:53.262468 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:53.262424    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7cca8a39-bee0-4705-bcde-b31060e44944-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gxcrm\" (UID: \"7cca8a39-bee0-4705-bcde-b31060e44944\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gxcrm"
Oct 30 13:02:53.262648 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:53.262627    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 13:02:53.262720 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:53.262696    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cca8a39-bee0-4705-bcde-b31060e44944-samples-operator-tls podName:7cca8a39-bee0-4705-bcde-b31060e44944 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:57.262677385 +0000 UTC m=+39.143912585 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7cca8a39-bee0-4705-bcde-b31060e44944-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-gxcrm" (UID: "7cca8a39-bee0-4705-bcde-b31060e44944") : secret "samples-operator-tls" not found
Oct 30 13:02:53.363927 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:53.363889    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-service-ca-bundle\") pod \"router-default-668b54766c-wgk28\" (UID: \"d5db3ba8-f1de-4fa5-9839-e7e8200da58f\") " pod="openshift-ingress/router-default-668b54766c-wgk28"
Oct 30 13:02:53.363927 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:53.363936    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-registry-tls\") pod \"image-registry-5b97bd7c47-dssmx\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") " pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:02:53.364273 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:53.363966    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-metrics-certs\") pod \"router-default-668b54766c-wgk28\" (UID: \"d5db3ba8-f1de-4fa5-9839-e7e8200da58f\") " pod="openshift-ingress/router-default-668b54766c-wgk28"
Oct 30 13:02:53.364273 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:53.364012    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-registry-tls\") pod \"image-registry-9677cf66b-kpdfl\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") " pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:02:53.364273 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:53.364178    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 13:02:53.364273 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:53.364195    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9677cf66b-kpdfl: secret "image-registry-tls" not found
Oct 30 13:02:53.364273 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:53.364258    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-registry-tls podName:b4fd1eb6-625a-4649-ba27-9e8798fd1bc5 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:57.364238922 +0000 UTC m=+39.245474125 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-registry-tls") pod "image-registry-9677cf66b-kpdfl" (UID: "b4fd1eb6-625a-4649-ba27-9e8798fd1bc5") : secret "image-registry-tls" not found
Oct 30 13:02:53.364853 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:53.364689    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-service-ca-bundle podName:d5db3ba8-f1de-4fa5-9839-e7e8200da58f nodeName:}" failed. No retries permitted until 2026-10-30 13:02:57.364671353 +0000 UTC m=+39.245906589 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-service-ca-bundle") pod "router-default-668b54766c-wgk28" (UID: "d5db3ba8-f1de-4fa5-9839-e7e8200da58f") : configmap references non-existent config key: service-ca.crt
Oct 30 13:02:53.364853 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:53.364796    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 13:02:53.364853 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:53.364808    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b97bd7c47-dssmx: secret "image-registry-tls" not found
Oct 30 13:02:53.364853 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:53.364844    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-registry-tls podName:feda4324-4a5f-491b-a04c-202c222261a0 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:57.364830527 +0000 UTC m=+39.246065726 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-registry-tls") pod "image-registry-5b97bd7c47-dssmx" (UID: "feda4324-4a5f-491b-a04c-202c222261a0") : secret "image-registry-tls" not found
Oct 30 13:02:53.365106 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:53.364906    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 13:02:53.365106 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:53.364940    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-metrics-certs podName:d5db3ba8-f1de-4fa5-9839-e7e8200da58f nodeName:}" failed. No retries permitted until 2026-10-30 13:02:57.364929886 +0000 UTC m=+39.246165085 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-metrics-certs") pod "router-default-668b54766c-wgk28" (UID: "d5db3ba8-f1de-4fa5-9839-e7e8200da58f") : secret "router-metrics-certs-default" not found
Oct 30 13:02:53.465100 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:53.465053    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e640effb-1ff4-40f2-8d7b-53732edcd2bf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-x8wqt\" (UID: \"e640effb-1ff4-40f2-8d7b-53732edcd2bf\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x8wqt"
Oct 30 13:02:53.465310 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:53.465153    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/63086243-e82d-4fc5-bfef-e4241c22ad4e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2w8jt\" (UID: \"63086243-e82d-4fc5-bfef-e4241c22ad4e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2w8jt"
Oct 30 13:02:53.465310 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:53.465254    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 13:02:53.465310 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:53.465285    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d0b2655-d52d-4c95-9589-d45237fa1057-metrics-tls\") pod \"dns-default-8nb7s\" (UID: \"0d0b2655-d52d-4c95-9589-d45237fa1057\") " pod="openshift-dns/dns-default-8nb7s"
Oct 30 13:02:53.465468 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:53.465330    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e640effb-1ff4-40f2-8d7b-53732edcd2bf-cluster-monitoring-operator-tls podName:e640effb-1ff4-40f2-8d7b-53732edcd2bf nodeName:}" failed. No retries permitted until 2026-10-30 13:02:57.465307635 +0000 UTC m=+39.346542833 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e640effb-1ff4-40f2-8d7b-53732edcd2bf-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-x8wqt" (UID: "e640effb-1ff4-40f2-8d7b-53732edcd2bf") : secret "cluster-monitoring-operator-tls" not found
Oct 30 13:02:53.465468 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:53.465384    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 13:02:53.465468 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:53.465452    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d0b2655-d52d-4c95-9589-d45237fa1057-metrics-tls podName:0d0b2655-d52d-4c95-9589-d45237fa1057 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:57.46542361 +0000 UTC m=+39.346658813 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0d0b2655-d52d-4c95-9589-d45237fa1057-metrics-tls") pod "dns-default-8nb7s" (UID: "0d0b2655-d52d-4c95-9589-d45237fa1057") : secret "dns-default-metrics-tls" not found
Oct 30 13:02:53.465617 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:53.465510    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 13:02:53.465617 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:53.465543    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63086243-e82d-4fc5-bfef-e4241c22ad4e-networking-console-plugin-cert podName:63086243-e82d-4fc5-bfef-e4241c22ad4e nodeName:}" failed. No retries permitted until 2026-10-30 13:02:57.465534959 +0000 UTC m=+39.346770174 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/63086243-e82d-4fc5-bfef-e4241c22ad4e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2w8jt" (UID: "63086243-e82d-4fc5-bfef-e4241c22ad4e") : secret "networking-console-plugin-cert" not found
Oct 30 13:02:54.073594 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:54.073369    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-lvf7h" podStartSLOduration=351.548654533 podStartE2EDuration="5m54.073349765s" totalImagesPullingTime="2.524695232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:54.071832209 +0000 UTC m=+35.953067432" watchObservedRunningTime="2026-10-30 13:02:54.073349765 +0000 UTC m=+35.954584990"
Oct 30 13:02:56.042174 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:56.042080    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jdj4_f93ee017-ab79-447c-90e0-5b85b71dba24/console-operator/0.log"
Oct 30 13:02:56.042968 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:56.042747    2576 generic.go:356] "Generic (PLEG): container finished" podID="f93ee017-ab79-447c-90e0-5b85b71dba24" containerID="545082067c6b7f15df136a65aab22d2e7bc78e4eb4bd8f7b0160306def1bbfda" exitCode=255
Oct 30 13:02:56.042968 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:56.042872    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-6jdj4" event={"ID":"f93ee017-ab79-447c-90e0-5b85b71dba24","Type":"ContainerDied","Data":"545082067c6b7f15df136a65aab22d2e7bc78e4eb4bd8f7b0160306def1bbfda"}
Oct 30 13:02:56.043154 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:56.043053    2576 scope.go:122] "RemoveContainer" containerID="545082067c6b7f15df136a65aab22d2e7bc78e4eb4bd8f7b0160306def1bbfda"
Oct 30 13:02:56.045476 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:56.045453    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-lgssx" event={"ID":"155bbdd4-b566-467f-88c6-ec5b4b881a09","Type":"ContainerStarted","Data":"2718d9cd1ed17a33c335f8c10461bb03121ca28199b5424a91e71e2b0303f8f9"}
Oct 30 13:02:56.095977 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:56.095918    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-nk2zz"]
Oct 30 13:02:56.112688 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:56.111439    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nk2zz"
Oct 30 13:02:56.119602 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:56.117789    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nk2zz"]
Oct 30 13:02:56.130212 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:56.126952    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 13:02:56.130212 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:56.126999    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fx9p2\""
Oct 30 13:02:56.130212 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:56.127751    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 13:02:56.130212 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:56.129362    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-lgssx" podStartSLOduration=347.385417256 podStartE2EDuration="5m52.129349697s" totalImagesPullingTime="4.743932441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:56.126579031 +0000 UTC m=+38.007814251" watchObservedRunningTime="2026-10-30 13:02:56.129349697 +0000 UTC m=+38.010584917"
Oct 30 13:02:56.160426 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:56.160386    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-9wlmq-z26dn"]
Oct 30 13:02:56.171218 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:56.171192    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-9wlmq-z26dn"
Oct 30 13:02:56.182892 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:56.180599    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-9wlmq-z26dn"]
Oct 30 13:02:56.210289 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:56.210201    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7ceb0bb1-def2-44a5-9e31-ceffd848125d-crio-socket\") pod \"insights-runtime-extractor-nk2zz\" (UID: \"7ceb0bb1-def2-44a5-9e31-ceffd848125d\") " pod="openshift-insights/insights-runtime-extractor-nk2zz"
Oct 30 13:02:56.211887 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:56.211855    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7ceb0bb1-def2-44a5-9e31-ceffd848125d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nk2zz\" (UID: \"7ceb0bb1-def2-44a5-9e31-ceffd848125d\") " pod="openshift-insights/insights-runtime-extractor-nk2zz"
Oct 30 13:02:56.212021 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:56.211911    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f4acf40a-d5b0-46e2-b87f-3b96fc9678c4-archives-path\") pod \"periodic-gathering-9wlmq-z26dn\" (UID: \"f4acf40a-d5b0-46e2-b87f-3b96fc9678c4\") " pod="openshift-insights/periodic-gathering-9wlmq-z26dn"
Oct 30 13:02:56.212021 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:56.211939    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4acf40a-d5b0-46e2-b87f-3b96fc9678c4-trusted-ca-bundle\") pod \"periodic-gathering-9wlmq-z26dn\" (UID: \"f4acf40a-d5b0-46e2-b87f-3b96fc9678c4\") " pod="openshift-insights/periodic-gathering-9wlmq-z26dn"
Oct 30 13:02:56.212160 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:56.212083    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4acf40a-d5b0-46e2-b87f-3b96fc9678c4-service-ca-bundle\") pod \"periodic-gathering-9wlmq-z26dn\" (UID: \"f4acf40a-d5b0-46e2-b87f-3b96fc9678c4\") " pod="openshift-insights/periodic-gathering-9wlmq-z26dn"
Oct 30 13:02:56.212160 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:56.212109    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7ceb0bb1-def2-44a5-9e31-ceffd848125d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nk2zz\" (UID: \"7ceb0bb1-def2-44a5-9e31-ceffd848125d\") " pod="openshift-insights/insights-runtime-extractor-nk2zz"
Oct 30 13:02:56.212263 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:56.212182    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdh8w\" (UniqueName: \"kubernetes.io/projected/f4acf40a-d5b0-46e2-b87f-3b96fc9678c4-kube-api-access-xdh8w\") pod \"periodic-gathering-9wlmq-z26dn\" (UID: \"f4acf40a-d5b0-46e2-b87f-3b96fc9678c4\") " pod="openshift-insights/periodic-gathering-9wlmq-z26dn"
Oct 30 13:02:56.212313 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:56.212250    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5k78\" (UniqueName: \"kubernetes.io/projected/7ceb0bb1-def2-44a5-9e31-ceffd848125d-kube-api-access-f5k78\") pod \"insights-runtime-extractor-nk2zz\" (UID: \"7ceb0bb1-def2-44a5-9e31-ceffd848125d\") " pod="openshift-insights/insights-runtime-extractor-nk2zz"
Oct 30 13:02:56.212365 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:56.212322    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ceb0bb1-def2-44a5-9e31-ceffd848125d-data-volume\") pod \"insights-runtime-extractor-nk2zz\" (UID: \"7ceb0bb1-def2-44a5-9e31-ceffd848125d\") " pod="openshift-insights/insights-runtime-extractor-nk2zz"
Oct 30 13:02:56.312955 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:56.312921    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdh8w\" (UniqueName: \"kubernetes.io/projected/f4acf40a-d5b0-46e2-b87f-3b96fc9678c4-kube-api-access-xdh8w\") pod \"periodic-gathering-9wlmq-z26dn\" (UID: \"f4acf40a-d5b0-46e2-b87f-3b96fc9678c4\") " pod="openshift-insights/periodic-gathering-9wlmq-z26dn"
Oct 30 13:02:56.313081 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:56.312974    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5k78\" (UniqueName: \"kubernetes.io/projected/7ceb0bb1-def2-44a5-9e31-ceffd848125d-kube-api-access-f5k78\") pod \"insights-runtime-extractor-nk2zz\" (UID: \"7ceb0bb1-def2-44a5-9e31-ceffd848125d\") " pod="openshift-insights/insights-runtime-extractor-nk2zz"
Oct 30 13:02:56.313081 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:56.312996    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ceb0bb1-def2-44a5-9e31-ceffd848125d-data-volume\") pod \"insights-runtime-extractor-nk2zz\" (UID: \"7ceb0bb1-def2-44a5-9e31-ceffd848125d\") " pod="openshift-insights/insights-runtime-extractor-nk2zz"
Oct 30 13:02:56.313081 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:56.313040    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7ceb0bb1-def2-44a5-9e31-ceffd848125d-crio-socket\") pod \"insights-runtime-extractor-nk2zz\" (UID: \"7ceb0bb1-def2-44a5-9e31-ceffd848125d\") " pod="openshift-insights/insights-runtime-extractor-nk2zz"
Oct 30 13:02:56.313317 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:56.313094    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7ceb0bb1-def2-44a5-9e31-ceffd848125d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nk2zz\" (UID: \"7ceb0bb1-def2-44a5-9e31-ceffd848125d\") " pod="openshift-insights/insights-runtime-extractor-nk2zz"
Oct 30 13:02:56.313317 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:56.313114    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f4acf40a-d5b0-46e2-b87f-3b96fc9678c4-archives-path\") pod \"periodic-gathering-9wlmq-z26dn\" (UID: \"f4acf40a-d5b0-46e2-b87f-3b96fc9678c4\") " pod="openshift-insights/periodic-gathering-9wlmq-z26dn"
Oct 30 13:02:56.313317 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:56.313143    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4acf40a-d5b0-46e2-b87f-3b96fc9678c4-trusted-ca-bundle\") pod \"periodic-gathering-9wlmq-z26dn\" (UID: \"f4acf40a-d5b0-46e2-b87f-3b96fc9678c4\") " pod="openshift-insights/periodic-gathering-9wlmq-z26dn"
Oct 30 13:02:56.313490 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:56.313472    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ceb0bb1-def2-44a5-9e31-ceffd848125d-data-volume\") pod \"insights-runtime-extractor-nk2zz\" (UID: \"7ceb0bb1-def2-44a5-9e31-ceffd848125d\") " pod="openshift-insights/insights-runtime-extractor-nk2zz"
Oct 30 13:02:56.314275 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:56.313662    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7ceb0bb1-def2-44a5-9e31-ceffd848125d-crio-socket\") pod \"insights-runtime-extractor-nk2zz\" (UID: \"7ceb0bb1-def2-44a5-9e31-ceffd848125d\") " pod="openshift-insights/insights-runtime-extractor-nk2zz"
Oct 30 13:02:56.314275 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:56.313687    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 13:02:56.314275 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:56.313760    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ceb0bb1-def2-44a5-9e31-ceffd848125d-insights-runtime-extractor-tls podName:7ceb0bb1-def2-44a5-9e31-ceffd848125d nodeName:}" failed. No retries permitted until 2026-10-30 13:02:56.813743605 +0000 UTC m=+38.694978839 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7ceb0bb1-def2-44a5-9e31-ceffd848125d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nk2zz" (UID: "7ceb0bb1-def2-44a5-9e31-ceffd848125d") : secret "insights-runtime-extractor-tls" not found
Oct 30 13:02:56.314275 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:56.314221    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f4acf40a-d5b0-46e2-b87f-3b96fc9678c4-archives-path\") pod \"periodic-gathering-9wlmq-z26dn\" (UID: \"f4acf40a-d5b0-46e2-b87f-3b96fc9678c4\") " pod="openshift-insights/periodic-gathering-9wlmq-z26dn"
Oct 30 13:02:56.314769 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:56.314749    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4acf40a-d5b0-46e2-b87f-3b96fc9678c4-trusted-ca-bundle\") pod \"periodic-gathering-9wlmq-z26dn\" (UID: \"f4acf40a-d5b0-46e2-b87f-3b96fc9678c4\") " pod="openshift-insights/periodic-gathering-9wlmq-z26dn"
Oct 30 13:02:56.314877 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:56.314856    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4acf40a-d5b0-46e2-b87f-3b96fc9678c4-service-ca-bundle\") pod \"periodic-gathering-9wlmq-z26dn\" (UID: \"f4acf40a-d5b0-46e2-b87f-3b96fc9678c4\") " pod="openshift-insights/periodic-gathering-9wlmq-z26dn"
Oct 30 13:02:56.314946 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:56.314885    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7ceb0bb1-def2-44a5-9e31-ceffd848125d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nk2zz\" (UID: \"7ceb0bb1-def2-44a5-9e31-ceffd848125d\") " pod="openshift-insights/insights-runtime-extractor-nk2zz"
Oct 30 13:02:56.315002 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:56.314957    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 13:02:56.315002 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:56.314989    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7ceb0bb1-def2-44a5-9e31-ceffd848125d-kube-rbac-proxy-cm podName:7ceb0bb1-def2-44a5-9e31-ceffd848125d nodeName:}" failed. No retries permitted until 2026-10-30 13:02:56.814980129 +0000 UTC m=+38.696215327 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7ceb0bb1-def2-44a5-9e31-ceffd848125d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nk2zz" (UID: "7ceb0bb1-def2-44a5-9e31-ceffd848125d") : configmap "kube-rbac-proxy" not found
Oct 30 13:02:56.315385 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:56.315367    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4acf40a-d5b0-46e2-b87f-3b96fc9678c4-service-ca-bundle\") pod \"periodic-gathering-9wlmq-z26dn\" (UID: \"f4acf40a-d5b0-46e2-b87f-3b96fc9678c4\") " pod="openshift-insights/periodic-gathering-9wlmq-z26dn"
Oct 30 13:02:56.333463 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:56.333434    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdh8w\" (UniqueName: \"kubernetes.io/projected/f4acf40a-d5b0-46e2-b87f-3b96fc9678c4-kube-api-access-xdh8w\") pod \"periodic-gathering-9wlmq-z26dn\" (UID: \"f4acf40a-d5b0-46e2-b87f-3b96fc9678c4\") " pod="openshift-insights/periodic-gathering-9wlmq-z26dn"
Oct 30 13:02:56.344842 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:56.344812    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5k78\" (UniqueName: \"kubernetes.io/projected/7ceb0bb1-def2-44a5-9e31-ceffd848125d-kube-api-access-f5k78\") pod \"insights-runtime-extractor-nk2zz\" (UID: \"7ceb0bb1-def2-44a5-9e31-ceffd848125d\") " pod="openshift-insights/insights-runtime-extractor-nk2zz"
Oct 30 13:02:56.566617 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:56.566509    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-9wlmq-z26dn"
Oct 30 13:02:56.820912 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:56.820828    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7ceb0bb1-def2-44a5-9e31-ceffd848125d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nk2zz\" (UID: \"7ceb0bb1-def2-44a5-9e31-ceffd848125d\") " pod="openshift-insights/insights-runtime-extractor-nk2zz"
Oct 30 13:02:56.821090 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:56.820913    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7ceb0bb1-def2-44a5-9e31-ceffd848125d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nk2zz\" (UID: \"7ceb0bb1-def2-44a5-9e31-ceffd848125d\") " pod="openshift-insights/insights-runtime-extractor-nk2zz"
Oct 30 13:02:56.821090 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:56.821047    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 13:02:56.821276 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:56.821111    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7ceb0bb1-def2-44a5-9e31-ceffd848125d-kube-rbac-proxy-cm podName:7ceb0bb1-def2-44a5-9e31-ceffd848125d nodeName:}" failed. No retries permitted until 2026-10-30 13:02:57.821092434 +0000 UTC m=+39.702327646 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7ceb0bb1-def2-44a5-9e31-ceffd848125d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nk2zz" (UID: "7ceb0bb1-def2-44a5-9e31-ceffd848125d") : configmap "kube-rbac-proxy" not found
Oct 30 13:02:56.821342 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:56.821286    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 13:02:56.821389 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:56.821356    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ceb0bb1-def2-44a5-9e31-ceffd848125d-insights-runtime-extractor-tls podName:7ceb0bb1-def2-44a5-9e31-ceffd848125d nodeName:}" failed. No retries permitted until 2026-10-30 13:02:57.821337863 +0000 UTC m=+39.702573066 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7ceb0bb1-def2-44a5-9e31-ceffd848125d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nk2zz" (UID: "7ceb0bb1-def2-44a5-9e31-ceffd848125d") : secret "insights-runtime-extractor-tls" not found
Oct 30 13:02:56.907361 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:56.907327    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-9wlmq-z26dn"]
Oct 30 13:02:57.056694 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:57.055901    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wmb4b" event={"ID":"1a25d5a7-83d4-443b-98ae-fc7b603cfc5f","Type":"ContainerStarted","Data":"12e15e534e5d6139172636de2562a1eeae816b34d36af63acfee49459234a3c9"}
Oct 30 13:02:57.065944 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:57.065369    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fhnzt" event={"ID":"79664154-5454-4294-baad-12bdfad6fe68","Type":"ContainerStarted","Data":"64af381430f07be2de9fc933146d3337b68583cf06e1405ad9f25c9aa87670bb"}
Oct 30 13:02:57.069100 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:57.069003    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jdj4_f93ee017-ab79-447c-90e0-5b85b71dba24/console-operator/1.log"
Oct 30 13:02:57.069614 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:57.069593    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jdj4_f93ee017-ab79-447c-90e0-5b85b71dba24/console-operator/0.log"
Oct 30 13:02:57.069713 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:57.069631    2576 generic.go:356] "Generic (PLEG): container finished" podID="f93ee017-ab79-447c-90e0-5b85b71dba24" containerID="843f19531f8ed50dca559ae32fd0ed454c48aa876303554a6b13044481fee1a7" exitCode=255
Oct 30 13:02:57.069713 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:57.069700    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-6jdj4" event={"ID":"f93ee017-ab79-447c-90e0-5b85b71dba24","Type":"ContainerDied","Data":"843f19531f8ed50dca559ae32fd0ed454c48aa876303554a6b13044481fee1a7"}
Oct 30 13:02:57.069778 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:57.069746    2576 scope.go:122] "RemoveContainer" containerID="545082067c6b7f15df136a65aab22d2e7bc78e4eb4bd8f7b0160306def1bbfda"
Oct 30 13:02:57.070176 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:57.070156    2576 scope.go:122] "RemoveContainer" containerID="843f19531f8ed50dca559ae32fd0ed454c48aa876303554a6b13044481fee1a7"
Oct 30 13:02:57.070782 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:57.070370    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-6jdj4_openshift-console-operator(f93ee017-ab79-447c-90e0-5b85b71dba24)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-6jdj4" podUID="f93ee017-ab79-447c-90e0-5b85b71dba24"
Oct 30 13:02:57.074533 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:57.074024    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lt2qg" event={"ID":"5cbda802-f819-4ea6-846e-440cea4afed2","Type":"ContainerStarted","Data":"0ec814bf9af33ffe20a9f56154cad2671a9939f1a592b0099e108dbf4e8fa5f7"}
Oct 30 13:02:57.077668 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:57.077603    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-9wlmq-z26dn" event={"ID":"f4acf40a-d5b0-46e2-b87f-3b96fc9678c4","Type":"ContainerStarted","Data":"3cc6fafce08f8f3d202026631327bb8e095f97ff186665b5b1996bd0fe1fbc6c"}
Oct 30 13:02:57.095644 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:57.095431    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wmb4b" podStartSLOduration=2.045637665 podStartE2EDuration="8.095416271s" totalImagesPullingTime="6.049778606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:57.093426097 +0000 UTC m=+38.974661316" watchObservedRunningTime="2026-10-30 13:02:57.095416271 +0000 UTC m=+38.976651490"
Oct 30 13:02:57.131493 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:57.130493    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fhnzt" podStartSLOduration=349.48976544 podStartE2EDuration="5m55.130474464s" totalImagesPullingTime="5.640709024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:57.129091861 +0000 UTC m=+39.010327082" watchObservedRunningTime="2026-10-30 13:02:57.130474464 +0000 UTC m=+39.011709684"
Oct 30 13:02:57.210250 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:57.209644    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lt2qg" podStartSLOduration=352.456401389 podStartE2EDuration="5m58.209623354s" totalImagesPullingTime="5.753221965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:56:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:57.207267545 +0000 UTC m=+39.088502766" watchObservedRunningTime="2026-10-30 13:02:57.209623354 +0000 UTC m=+39.090858575"
Oct 30 13:02:57.247013 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:57.246187    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-9wlmq-z26dn" podStartSLOduration=1.246166607 podStartE2EDuration="1.246166607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:57.244801299 +0000 UTC m=+39.126036544" watchObservedRunningTime="2026-10-30 13:02:57.246166607 +0000 UTC m=+39.127401828"
Oct 30 13:02:57.325889 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:57.325757    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7cca8a39-bee0-4705-bcde-b31060e44944-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gxcrm\" (UID: \"7cca8a39-bee0-4705-bcde-b31060e44944\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gxcrm"
Oct 30 13:02:57.326076 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:57.325957    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 13:02:57.326076 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:57.326043    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cca8a39-bee0-4705-bcde-b31060e44944-samples-operator-tls podName:7cca8a39-bee0-4705-bcde-b31060e44944 nodeName:}" failed. No retries permitted until 2026-10-30 13:03:05.32601921 +0000 UTC m=+47.207254428 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7cca8a39-bee0-4705-bcde-b31060e44944-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-gxcrm" (UID: "7cca8a39-bee0-4705-bcde-b31060e44944") : secret "samples-operator-tls" not found
Oct 30 13:02:57.427094 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:57.427053    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-service-ca-bundle\") pod \"router-default-668b54766c-wgk28\" (UID: \"d5db3ba8-f1de-4fa5-9839-e7e8200da58f\") " pod="openshift-ingress/router-default-668b54766c-wgk28"
Oct 30 13:02:57.427277 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:57.427109    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-registry-tls\") pod \"image-registry-5b97bd7c47-dssmx\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") " pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:02:57.427277 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:57.427161    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-metrics-certs\") pod \"router-default-668b54766c-wgk28\" (UID: \"d5db3ba8-f1de-4fa5-9839-e7e8200da58f\") " pod="openshift-ingress/router-default-668b54766c-wgk28"
Oct 30 13:02:57.427277 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:57.427229    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-service-ca-bundle podName:d5db3ba8-f1de-4fa5-9839-e7e8200da58f nodeName:}" failed. No retries permitted until 2026-10-30 13:03:05.427209953 +0000 UTC m=+47.308445172 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-service-ca-bundle") pod "router-default-668b54766c-wgk28" (UID: "d5db3ba8-f1de-4fa5-9839-e7e8200da58f") : configmap references non-existent config key: service-ca.crt
Oct 30 13:02:57.427277 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:57.427262    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 13:02:57.427277 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:57.427273    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-registry-tls\") pod \"image-registry-9677cf66b-kpdfl\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") " pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:02:57.427474 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:57.427283    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b97bd7c47-dssmx: secret "image-registry-tls" not found
Oct 30 13:02:57.427474 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:57.427284    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 13:02:57.427474 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:57.427332    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-registry-tls podName:feda4324-4a5f-491b-a04c-202c222261a0 nodeName:}" failed. No retries permitted until 2026-10-30 13:03:05.427317841 +0000 UTC m=+47.308553088 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-registry-tls") pod "image-registry-5b97bd7c47-dssmx" (UID: "feda4324-4a5f-491b-a04c-202c222261a0") : secret "image-registry-tls" not found
Oct 30 13:02:57.427474 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:57.427346    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-metrics-certs podName:d5db3ba8-f1de-4fa5-9839-e7e8200da58f nodeName:}" failed. No retries permitted until 2026-10-30 13:03:05.427340203 +0000 UTC m=+47.308575401 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-metrics-certs") pod "router-default-668b54766c-wgk28" (UID: "d5db3ba8-f1de-4fa5-9839-e7e8200da58f") : secret "router-metrics-certs-default" not found
Oct 30 13:02:57.427474 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:57.427367    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 13:02:57.427474 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:57.427377    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9677cf66b-kpdfl: secret "image-registry-tls" not found
Oct 30 13:02:57.427474 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:57.427430    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-registry-tls podName:b4fd1eb6-625a-4649-ba27-9e8798fd1bc5 nodeName:}" failed. No retries permitted until 2026-10-30 13:03:05.42741818 +0000 UTC m=+47.308653406 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-registry-tls") pod "image-registry-9677cf66b-kpdfl" (UID: "b4fd1eb6-625a-4649-ba27-9e8798fd1bc5") : secret "image-registry-tls" not found
Oct 30 13:02:57.487224 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:57.487189    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-k6rlj"]
Oct 30 13:02:57.501323 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:57.501292    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-r5pcv"]
Oct 30 13:02:57.501510 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:57.501434    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-k6rlj"
Oct 30 13:02:57.515157 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:57.515114    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-k6rlj"]
Oct 30 13:02:57.515334 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:57.515239    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-r5pcv"
Oct 30 13:02:57.522162 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:57.521861    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-r5pcv"]
Oct 30 13:02:57.523000 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:57.522916    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-vqv56\""
Oct 30 13:02:57.531532 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:57.527719    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e640effb-1ff4-40f2-8d7b-53732edcd2bf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-x8wqt\" (UID: \"e640effb-1ff4-40f2-8d7b-53732edcd2bf\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x8wqt"
Oct 30 13:02:57.531532 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:57.527783    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/63086243-e82d-4fc5-bfef-e4241c22ad4e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2w8jt\" (UID: \"63086243-e82d-4fc5-bfef-e4241c22ad4e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2w8jt"
Oct 30 13:02:57.531532 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:57.527833    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4kjx\" (UniqueName: \"kubernetes.io/projected/3c7ab011-8fd4-4d6d-b3f2-72d6af11d3e1-kube-api-access-p4kjx\") pod \"migrator-66df5d496b-k6rlj\" (UID: \"3c7ab011-8fd4-4d6d-b3f2-72d6af11d3e1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-k6rlj"
Oct 30 13:02:57.531532 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:57.527977    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d0b2655-d52d-4c95-9589-d45237fa1057-metrics-tls\") pod \"dns-default-8nb7s\" (UID: \"0d0b2655-d52d-4c95-9589-d45237fa1057\") " pod="openshift-dns/dns-default-8nb7s"
Oct 30 13:02:57.531532 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:57.528174    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 13:02:57.531532 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:57.528237    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d0b2655-d52d-4c95-9589-d45237fa1057-metrics-tls podName:0d0b2655-d52d-4c95-9589-d45237fa1057 nodeName:}" failed. No retries permitted until 2026-10-30 13:03:05.528219707 +0000 UTC m=+47.409454927 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0d0b2655-d52d-4c95-9589-d45237fa1057-metrics-tls") pod "dns-default-8nb7s" (UID: "0d0b2655-d52d-4c95-9589-d45237fa1057") : secret "dns-default-metrics-tls" not found
Oct 30 13:02:57.531532 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:57.528616    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 13:02:57.531532 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:57.528676    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e640effb-1ff4-40f2-8d7b-53732edcd2bf-cluster-monitoring-operator-tls podName:e640effb-1ff4-40f2-8d7b-53732edcd2bf nodeName:}" failed. No retries permitted until 2026-10-30 13:03:05.528657545 +0000 UTC m=+47.409892758 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e640effb-1ff4-40f2-8d7b-53732edcd2bf-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-x8wqt" (UID: "e640effb-1ff4-40f2-8d7b-53732edcd2bf") : secret "cluster-monitoring-operator-tls" not found
Oct 30 13:02:57.531532 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:57.528733    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 13:02:57.531532 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:57.528770    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63086243-e82d-4fc5-bfef-e4241c22ad4e-networking-console-plugin-cert podName:63086243-e82d-4fc5-bfef-e4241c22ad4e nodeName:}" failed. No retries permitted until 2026-10-30 13:03:05.528758409 +0000 UTC m=+47.409993625 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/63086243-e82d-4fc5-bfef-e4241c22ad4e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2w8jt" (UID: "63086243-e82d-4fc5-bfef-e4241c22ad4e") : secret "networking-console-plugin-cert" not found
Oct 30 13:02:57.535415 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:57.535389    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 13:02:57.536067 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:57.536043    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 13:02:57.629627 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:57.629541    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4kjx\" (UniqueName: \"kubernetes.io/projected/3c7ab011-8fd4-4d6d-b3f2-72d6af11d3e1-kube-api-access-p4kjx\") pod \"migrator-66df5d496b-k6rlj\" (UID: \"3c7ab011-8fd4-4d6d-b3f2-72d6af11d3e1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-k6rlj"
Oct 30 13:02:57.629798 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:57.629683    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jdqx\" (UniqueName: \"kubernetes.io/projected/7bbbc7e0-1790-459b-952a-b6b7b4d55113-kube-api-access-8jdqx\") pod \"migrator-66df5d496b-r5pcv\" (UID: \"7bbbc7e0-1790-459b-952a-b6b7b4d55113\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-r5pcv"
Oct 30 13:02:57.647868 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:57.647839    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4kjx\" (UniqueName: \"kubernetes.io/projected/3c7ab011-8fd4-4d6d-b3f2-72d6af11d3e1-kube-api-access-p4kjx\") pod \"migrator-66df5d496b-k6rlj\" (UID: \"3c7ab011-8fd4-4d6d-b3f2-72d6af11d3e1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-k6rlj"
Oct 30 13:02:57.730890 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:57.730853    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jdqx\" (UniqueName: \"kubernetes.io/projected/7bbbc7e0-1790-459b-952a-b6b7b4d55113-kube-api-access-8jdqx\") pod \"migrator-66df5d496b-r5pcv\" (UID: \"7bbbc7e0-1790-459b-952a-b6b7b4d55113\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-r5pcv"
Oct 30 13:02:57.750809 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:57.750745    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jdqx\" (UniqueName: \"kubernetes.io/projected/7bbbc7e0-1790-459b-952a-b6b7b4d55113-kube-api-access-8jdqx\") pod \"migrator-66df5d496b-r5pcv\" (UID: \"7bbbc7e0-1790-459b-952a-b6b7b4d55113\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-r5pcv"
Oct 30 13:02:57.831866 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:57.831832    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7ceb0bb1-def2-44a5-9e31-ceffd848125d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nk2zz\" (UID: \"7ceb0bb1-def2-44a5-9e31-ceffd848125d\") " pod="openshift-insights/insights-runtime-extractor-nk2zz"
Oct 30 13:02:57.832058 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:57.831918    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7ceb0bb1-def2-44a5-9e31-ceffd848125d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nk2zz\" (UID: \"7ceb0bb1-def2-44a5-9e31-ceffd848125d\") " pod="openshift-insights/insights-runtime-extractor-nk2zz"
Oct 30 13:02:57.832058 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:57.832005    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-k6rlj"
Oct 30 13:02:57.832379 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:57.832009    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 13:02:57.832379 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:57.832187    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7ceb0bb1-def2-44a5-9e31-ceffd848125d-kube-rbac-proxy-cm podName:7ceb0bb1-def2-44a5-9e31-ceffd848125d nodeName:}" failed. No retries permitted until 2026-10-30 13:02:59.832164265 +0000 UTC m=+41.713399481 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7ceb0bb1-def2-44a5-9e31-ceffd848125d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nk2zz" (UID: "7ceb0bb1-def2-44a5-9e31-ceffd848125d") : configmap "kube-rbac-proxy" not found
Oct 30 13:02:57.832379 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:57.832016    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 13:02:57.832379 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:57.832323    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ceb0bb1-def2-44a5-9e31-ceffd848125d-insights-runtime-extractor-tls podName:7ceb0bb1-def2-44a5-9e31-ceffd848125d nodeName:}" failed. No retries permitted until 2026-10-30 13:02:59.832303805 +0000 UTC m=+41.713539004 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7ceb0bb1-def2-44a5-9e31-ceffd848125d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nk2zz" (UID: "7ceb0bb1-def2-44a5-9e31-ceffd848125d") : secret "insights-runtime-extractor-tls" not found
Oct 30 13:02:57.853267 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:57.853241    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-r5pcv"
Oct 30 13:02:58.010415 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:58.010380    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-x8ck2"]
Oct 30 13:02:58.043915 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:58.043224    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-x8ck2"]
Oct 30 13:02:58.043915 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:58.043366    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-x8ck2"
Oct 30 13:02:58.057359 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:58.057263    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 13:02:58.057359 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:58.057291    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-w2zkq\""
Oct 30 13:02:58.057359 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:58.057275    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 13:02:58.057359 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:58.057325    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 13:02:58.057359 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:58.057334    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 13:02:58.084259 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:58.084167    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-r7vrw" event={"ID":"4ad66a0c-8dc0-40a4-a1b2-28d26b0de94b","Type":"ContainerStarted","Data":"49ddf73c0dd8f107c469e1493c8208bc951c2011938be8bf600c437065e440c4"}
Oct 30 13:02:58.086064 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:58.086039    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jdj4_f93ee017-ab79-447c-90e0-5b85b71dba24/console-operator/1.log"
Oct 30 13:02:58.086445 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:58.086424    2576 scope.go:122] "RemoveContainer" containerID="843f19531f8ed50dca559ae32fd0ed454c48aa876303554a6b13044481fee1a7"
Oct 30 13:02:58.086803 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:58.086620    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-6jdj4_openshift-console-operator(f93ee017-ab79-447c-90e0-5b85b71dba24)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-6jdj4" podUID="f93ee017-ab79-447c-90e0-5b85b71dba24"
Oct 30 13:02:58.087995 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:58.087967    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-9wlmq-z26dn" event={"ID":"f4acf40a-d5b0-46e2-b87f-3b96fc9678c4","Type":"ContainerStarted","Data":"88919911d03e7a59c8ca765bf094fd152dbb01b64488b1c70c627d30991354ed"}
Oct 30 13:02:58.134911 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:58.134827    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-789vz\" (UniqueName: \"kubernetes.io/projected/7d29458a-da6c-461c-9941-afa323061f30-kube-api-access-789vz\") pod \"service-ca-5fffc97bdf-x8ck2\" (UID: \"7d29458a-da6c-461c-9941-afa323061f30\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x8ck2"
Oct 30 13:02:58.135056 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:58.135035    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7d29458a-da6c-461c-9941-afa323061f30-signing-cabundle\") pod \"service-ca-5fffc97bdf-x8ck2\" (UID: \"7d29458a-da6c-461c-9941-afa323061f30\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x8ck2"
Oct 30 13:02:58.135405 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:58.135382    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7d29458a-da6c-461c-9941-afa323061f30-signing-key\") pod \"service-ca-5fffc97bdf-x8ck2\" (UID: \"7d29458a-da6c-461c-9941-afa323061f30\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x8ck2"
Oct 30 13:02:58.158497 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:58.158442    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-r7vrw" podStartSLOduration=344.944482632 podStartE2EDuration="5m52.158426534s" totalImagesPullingTime="7.213943902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:58.118940307 +0000 UTC m=+40.000175531" watchObservedRunningTime="2026-10-30 13:02:58.158426534 +0000 UTC m=+40.039661755"
Oct 30 13:02:58.161647 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:58.161575    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-k6rlj"]
Oct 30 13:02:58.174644 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:58.174605    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c7ab011_8fd4_4d6d_b3f2_72d6af11d3e1.slice/crio-f227bcba4afc53a24a6c3631b7e33b3cc84b82b155f54df82731be6b10bc97fc WatchSource:0}: Error finding container f227bcba4afc53a24a6c3631b7e33b3cc84b82b155f54df82731be6b10bc97fc: Status 404 returned error can't find the container with id f227bcba4afc53a24a6c3631b7e33b3cc84b82b155f54df82731be6b10bc97fc
Oct 30 13:02:58.195203 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:58.195161    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-r5pcv"]
Oct 30 13:02:58.200089 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:58.200057    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bbbc7e0_1790_459b_952a_b6b7b4d55113.slice/crio-09db080f67588acba375556f890373b7865b100a2a41cf985e15ec97b6c1899b WatchSource:0}: Error finding container 09db080f67588acba375556f890373b7865b100a2a41cf985e15ec97b6c1899b: Status 404 returned error can't find the container with id 09db080f67588acba375556f890373b7865b100a2a41cf985e15ec97b6c1899b
Oct 30 13:02:58.235973 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:58.235946    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7d29458a-da6c-461c-9941-afa323061f30-signing-key\") pod \"service-ca-5fffc97bdf-x8ck2\" (UID: \"7d29458a-da6c-461c-9941-afa323061f30\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x8ck2"
Oct 30 13:02:58.236086 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:58.236032    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-789vz\" (UniqueName: \"kubernetes.io/projected/7d29458a-da6c-461c-9941-afa323061f30-kube-api-access-789vz\") pod \"service-ca-5fffc97bdf-x8ck2\" (UID: \"7d29458a-da6c-461c-9941-afa323061f30\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x8ck2"
Oct 30 13:02:58.236159 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:58.236102    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7d29458a-da6c-461c-9941-afa323061f30-signing-cabundle\") pod \"service-ca-5fffc97bdf-x8ck2\" (UID: \"7d29458a-da6c-461c-9941-afa323061f30\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x8ck2"
Oct 30 13:02:58.237494 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:58.237437    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7d29458a-da6c-461c-9941-afa323061f30-signing-cabundle\") pod \"service-ca-5fffc97bdf-x8ck2\" (UID: \"7d29458a-da6c-461c-9941-afa323061f30\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x8ck2"
Oct 30 13:02:58.239110 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:58.239089    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7d29458a-da6c-461c-9941-afa323061f30-signing-key\") pod \"service-ca-5fffc97bdf-x8ck2\" (UID: \"7d29458a-da6c-461c-9941-afa323061f30\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x8ck2"
Oct 30 13:02:58.259349 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:58.257829    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-789vz\" (UniqueName: \"kubernetes.io/projected/7d29458a-da6c-461c-9941-afa323061f30-kube-api-access-789vz\") pod \"service-ca-5fffc97bdf-x8ck2\" (UID: \"7d29458a-da6c-461c-9941-afa323061f30\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x8ck2"
Oct 30 13:02:58.374516 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:58.374477    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-x8ck2"
Oct 30 13:02:58.684855 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:58.684810    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-x8ck2"]
Oct 30 13:02:58.690516 ip-10-0-134-31 kubenswrapper[2576]: W1030 13:02:58.690479    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d29458a_da6c_461c_9941_afa323061f30.slice/crio-dfe8cfde85a2c351da022ee8103df05f789b03b6ed99ffc5b984323d5cac5816 WatchSource:0}: Error finding container dfe8cfde85a2c351da022ee8103df05f789b03b6ed99ffc5b984323d5cac5816: Status 404 returned error can't find the container with id dfe8cfde85a2c351da022ee8103df05f789b03b6ed99ffc5b984323d5cac5816
Oct 30 13:02:58.921149 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:58.920967    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gltng_5a16e7b2-e45d-480a-a7bc-d914640b9867/dns-node-resolver/0.log"
Oct 30 13:02:59.104282 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:59.104200    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-k6rlj" event={"ID":"3c7ab011-8fd4-4d6d-b3f2-72d6af11d3e1","Type":"ContainerStarted","Data":"f227bcba4afc53a24a6c3631b7e33b3cc84b82b155f54df82731be6b10bc97fc"}
Oct 30 13:02:59.109335 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:59.109303    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rxd99" event={"ID":"a07f5a09-920e-462f-ab6d-7a2ac53d199e","Type":"ContainerStarted","Data":"0f21b12b434e7a6285128102ff4b343e77670be25b8b449d34950625d7e94077"}
Oct 30 13:02:59.109465 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:59.109386    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rxd99"
Oct 30 13:02:59.110982 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:59.110910    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-r5pcv" event={"ID":"7bbbc7e0-1790-459b-952a-b6b7b4d55113","Type":"ContainerStarted","Data":"09db080f67588acba375556f890373b7865b100a2a41cf985e15ec97b6c1899b"}
Oct 30 13:02:59.113069 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:59.113038    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-x8ck2" event={"ID":"7d29458a-da6c-461c-9941-afa323061f30","Type":"ContainerStarted","Data":"edae2073d643c6290e248e6129cd4ca6f8fce4f481d02c7d52ff1cda49281de1"}
Oct 30 13:02:59.113202 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:59.113090    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-x8ck2" event={"ID":"7d29458a-da6c-461c-9941-afa323061f30","Type":"ContainerStarted","Data":"dfe8cfde85a2c351da022ee8103df05f789b03b6ed99ffc5b984323d5cac5816"}
Oct 30 13:02:59.151037 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:59.150973    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rxd99" podStartSLOduration=34.067312599 podStartE2EDuration="40.15095376s" totalImagesPullingTime="6.083641161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:59.149900718 +0000 UTC m=+41.031135940" watchObservedRunningTime="2026-10-30 13:02:59.15095376 +0000 UTC m=+41.032188981"
Oct 30 13:02:59.188739 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:59.188673    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-x8ck2" podStartSLOduration=2.18865336 podStartE2EDuration="2.18865336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:59.187270943 +0000 UTC m=+41.068506174" watchObservedRunningTime="2026-10-30 13:02:59.18865336 +0000 UTC m=+41.069888580"
Oct 30 13:02:59.720699 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:59.720662    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-f7mvz_087058f5-e286-4ac7-8ed8-a280da332878/node-ca/0.log"
Oct 30 13:02:59.865152 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:59.861829    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7ceb0bb1-def2-44a5-9e31-ceffd848125d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nk2zz\" (UID: \"7ceb0bb1-def2-44a5-9e31-ceffd848125d\") " pod="openshift-insights/insights-runtime-extractor-nk2zz"
Oct 30 13:02:59.865152 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:59.861914    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7ceb0bb1-def2-44a5-9e31-ceffd848125d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nk2zz\" (UID: \"7ceb0bb1-def2-44a5-9e31-ceffd848125d\") " pod="openshift-insights/insights-runtime-extractor-nk2zz"
Oct 30 13:02:59.865152 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:59.861967    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 13:02:59.865152 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:02:59.862028    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ceb0bb1-def2-44a5-9e31-ceffd848125d-insights-runtime-extractor-tls podName:7ceb0bb1-def2-44a5-9e31-ceffd848125d nodeName:}" failed. No retries permitted until 2026-10-30 13:03:03.862012639 +0000 UTC m=+45.743247841 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7ceb0bb1-def2-44a5-9e31-ceffd848125d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nk2zz" (UID: "7ceb0bb1-def2-44a5-9e31-ceffd848125d") : secret "insights-runtime-extractor-tls" not found
Oct 30 13:02:59.865152 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:02:59.862442    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7ceb0bb1-def2-44a5-9e31-ceffd848125d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nk2zz\" (UID: \"7ceb0bb1-def2-44a5-9e31-ceffd848125d\") " pod="openshift-insights/insights-runtime-extractor-nk2zz"
Oct 30 13:03:00.085450 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:03:00.085425    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-6jdj4"
Oct 30 13:03:00.085557 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:03:00.085485    2576 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-6jdj4"
Oct 30 13:03:00.085912 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:03:00.085899    2576 scope.go:122] "RemoveContainer" containerID="843f19531f8ed50dca559ae32fd0ed454c48aa876303554a6b13044481fee1a7"
Oct 30 13:03:00.086157 ip-10-0-134-31 kubenswrapper[2576]: E1030 13:03:00.086119    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-6jdj4_openshift-console-operator(f93ee017-ab79-447c-90e0-5b85b71dba24)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-6jdj4" podUID="f93ee017-ab79-447c-90e0-5b85b71dba24"
Oct 30 13:03:00.326523 ip-10-0-134-31 kubenswrapper[2576]: I1030 13:03:00.326446    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wmb4b_1a25d5a7-83d4-443b-98ae-fc7b603cfc5f/serve-healthcheck-canary/0.log"
Oct 30 13:03:00.786359 ip-10-0-134-31 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 13:03:00.802554 ip-10-0-134-31 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 13:03:00.802739 ip-10-0-134-31 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 13:03:00.802789 ip-10-0-134-31 systemd[1]: kubelet.service: Consumed 3.337s CPU time.
Oct 30 13:03:00.820654 ip-10-0-134-31 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 13:03:00.902568 ip-10-0-134-31 kubenswrapper[5093]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 13:03:00.902568 ip-10-0-134-31 kubenswrapper[5093]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 13:03:00.902568 ip-10-0-134-31 kubenswrapper[5093]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 13:03:00.902568 ip-10-0-134-31 kubenswrapper[5093]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 13:03:00.904861 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.904763    5093 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 13:03:00.904861 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.904856    5093 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 13:03:00.904861 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.904861    5093 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 13:03:00.904861 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.904865    5093 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 13:03:00.904861 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.904870    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 13:03:00.905216 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.904875    5093 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 13:03:00.905216 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.904879    5093 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 13:03:00.905216 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.904883    5093 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 13:03:00.905216 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.904887    5093 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 13:03:00.905216 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.904890    5093 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 13:03:00.905216 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.904894    5093 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 13:03:00.905216 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.904898    5093 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 13:03:00.905216 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.904901    5093 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 13:03:00.905216 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.904905    5093 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 13:03:00.905216 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.904909    5093 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 13:03:00.905216 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.904914    5093 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 13:03:00.905216 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.904918    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 13:03:00.905216 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.904921    5093 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 13:03:00.905216 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.904925    5093 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 13:03:00.905216 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.904929    5093 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 13:03:00.905216 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.904933    5093 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 13:03:00.905216 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.904939    5093 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 13:03:00.905216 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.904946    5093 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 13:03:00.905216 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.904950    5093 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 13:03:00.906043 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.904954    5093 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 13:03:00.906043 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.904958    5093 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 13:03:00.906043 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.904962    5093 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 13:03:00.906043 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.904976    5093 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 13:03:00.906043 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.904980    5093 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 13:03:00.906043 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.904984    5093 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 13:03:00.906043 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.904988    5093 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 13:03:00.906043 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.904992    5093 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 13:03:00.906043 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.904997    5093 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 13:03:00.906043 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905001    5093 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 13:03:00.906043 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905005    5093 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 13:03:00.906043 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905009    5093 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 13:03:00.906043 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905013    5093 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 13:03:00.906043 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905017    5093 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 13:03:00.906043 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905023    5093 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 13:03:00.906043 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905027    5093 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 13:03:00.906043 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905032    5093 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 13:03:00.906043 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905036    5093 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 13:03:00.906043 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905040    5093 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 13:03:00.906043 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905044    5093 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 13:03:00.906919 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905051    5093 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 13:03:00.906919 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905055    5093 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 13:03:00.906919 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905060    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 13:03:00.906919 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905066    5093 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 13:03:00.906919 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905073    5093 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 13:03:00.906919 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905077    5093 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 13:03:00.906919 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905082    5093 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 13:03:00.906919 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905086    5093 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 13:03:00.906919 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905090    5093 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 13:03:00.906919 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905094    5093 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 13:03:00.906919 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905099    5093 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 13:03:00.906919 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905103    5093 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 13:03:00.906919 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905107    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 13:03:00.906919 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905112    5093 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 13:03:00.906919 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905116    5093 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 13:03:00.906919 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905121    5093 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 13:03:00.906919 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905142    5093 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 13:03:00.906919 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905147    5093 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 13:03:00.906919 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905152    5093 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 13:03:00.907780 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905157    5093 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 13:03:00.907780 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905161    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 13:03:00.907780 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905165    5093 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 13:03:00.907780 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905169    5093 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 13:03:00.907780 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905173    5093 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 13:03:00.907780 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905177    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 13:03:00.907780 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905181    5093 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 13:03:00.907780 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905187    5093 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 13:03:00.907780 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905192    5093 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 13:03:00.907780 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905196    5093 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 13:03:00.907780 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905201    5093 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 13:03:00.907780 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905206    5093 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 13:03:00.907780 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905210    5093 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 13:03:00.907780 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905214    5093 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 13:03:00.907780 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905218    5093 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 13:03:00.907780 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905222    5093 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 13:03:00.907780 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905227    5093 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 13:03:00.907780 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905231    5093 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 13:03:00.907780 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905235    5093 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 13:03:00.907780 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905240    5093 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 13:03:00.908678 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905244    5093 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 13:03:00.908678 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905249    5093 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 13:03:00.908678 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905253    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 13:03:00.908678 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905257    5093 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 13:03:00.908678 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905261    5093 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 13:03:00.908678 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905269    5093 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 13:03:00.908678 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905273    5093 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 13:03:00.908678 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905278    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 13:03:00.908678 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905282    5093 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 13:03:00.908678 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905287    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 13:03:00.908678 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905296    5093 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 13:03:00.908678 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905302    5093 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 13:03:00.908678 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905306    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 13:03:00.908678 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905310    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 13:03:00.908678 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905314    5093 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 13:03:00.908678 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905319    5093 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 13:03:00.908678 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905323    5093 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 13:03:00.908678 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905327    5093 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 13:03:00.908678 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905332    5093 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 13:03:00.909571 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905336    5093 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 13:03:00.909571 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905341    5093 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 13:03:00.909571 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905345    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 13:03:00.909571 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.905350    5093 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 13:03:00.909571 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906633    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 13:03:00.909571 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906644    5093 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 13:03:00.909571 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906648    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 13:03:00.909571 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906653    5093 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 13:03:00.909571 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906657    5093 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 13:03:00.909571 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906661    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 13:03:00.909571 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906666    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 13:03:00.909571 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906670    5093 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 13:03:00.909571 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906674    5093 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 13:03:00.909571 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906679    5093 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 13:03:00.909571 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906683    5093 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 13:03:00.909571 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906687    5093 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 13:03:00.909571 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906691    5093 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 13:03:00.909571 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906695    5093 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 13:03:00.909571 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906699    5093 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 13:03:00.909571 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906704    5093 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 13:03:00.910469 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906709    5093 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 13:03:00.910469 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906713    5093 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 13:03:00.910469 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906718    5093 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 13:03:00.910469 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906722    5093 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 13:03:00.910469 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906734    5093 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 13:03:00.910469 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906739    5093 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 13:03:00.910469 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906743    5093 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 13:03:00.910469 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906748    5093 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 13:03:00.910469 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906752    5093 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 13:03:00.910469 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906756    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 13:03:00.910469 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906760    5093 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 13:03:00.910469 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906765    5093 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 13:03:00.910469 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906769    5093 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 13:03:00.910469 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906773    5093 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 13:03:00.910469 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906778    5093 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 13:03:00.910469 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906782    5093 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 13:03:00.910469 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906786    5093 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 13:03:00.910469 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906791    5093 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 13:03:00.910469 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906795    5093 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 13:03:00.911281 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906800    5093 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 13:03:00.911281 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906804    5093 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 13:03:00.911281 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906812    5093 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 13:03:00.911281 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906819    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 13:03:00.911281 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906826    5093 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 13:03:00.911281 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906831    5093 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 13:03:00.911281 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906835    5093 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 13:03:00.911281 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906839    5093 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 13:03:00.911281 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906844    5093 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 13:03:00.911281 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906848    5093 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 13:03:00.911281 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906852    5093 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 13:03:00.911281 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906857    5093 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 13:03:00.911281 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906861    5093 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 13:03:00.911281 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906865    5093 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 13:03:00.911281 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906870    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 13:03:00.911281 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906874    5093 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 13:03:00.911281 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906879    5093 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 13:03:00.911281 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906883    5093 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 13:03:00.911281 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906895    5093 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 13:03:00.911848 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906900    5093 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 13:03:00.911848 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906904    5093 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 13:03:00.911848 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906909    5093 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 13:03:00.911848 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906913    5093 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 13:03:00.911848 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906917    5093 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 13:03:00.911848 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906921    5093 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 13:03:00.911848 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906927    5093 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 13:03:00.911848 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906931    5093 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 13:03:00.911848 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906935    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 13:03:00.911848 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906939    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 13:03:00.911848 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906943    5093 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 13:03:00.911848 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906947    5093 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 13:03:00.911848 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906951    5093 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 13:03:00.911848 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906956    5093 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 13:03:00.911848 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906962    5093 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 13:03:00.911848 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906968    5093 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 13:03:00.911848 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906973    5093 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 13:03:00.911848 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906978    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 13:03:00.911848 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906982    5093 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 13:03:00.912363 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906986    5093 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 13:03:00.912363 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906991    5093 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 13:03:00.912363 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906995    5093 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 13:03:00.912363 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.906999    5093 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 13:03:00.912363 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.907004    5093 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 13:03:00.912363 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.907008    5093 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 13:03:00.912363 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.907012    5093 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 13:03:00.912363 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.907017    5093 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 13:03:00.912363 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.907022    5093 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 13:03:00.912363 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.907026    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 13:03:00.912363 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.907044    5093 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 13:03:00.912363 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.907049    5093 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 13:03:00.912363 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.907053    5093 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 13:03:00.912363 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.907065    5093 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 13:03:00.912363 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.907071    5093 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 13:03:00.912363 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.907075    5093 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 13:03:00.912363 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.907080    5093 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 13:03:00.912363 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.907084    5093 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 13:03:00.912363 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.907088    5093 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 13:03:00.912363 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.907093    5093 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 13:03:00.912881 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.907097    5093 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 13:03:00.912881 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.907101    5093 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 13:03:00.912881 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.907105    5093 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 13:03:00.912881 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.907109    5093 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 13:03:00.912881 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.907113    5093 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 13:03:00.912881 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.907118    5093 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 13:03:00.912881 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.907122    5093 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 13:03:00.912881 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.907142    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 13:03:00.912881 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.907147    5093 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 13:03:00.912881 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.907152    5093 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 13:03:00.912881 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.907156    5093 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 13:03:00.912881 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.907160    5093 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 13:03:00.912881 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.907165    5093 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 13:03:00.912881 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.907977    5093 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 13:03:00.912881 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.907986    5093 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 13:03:00.912881 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.907990    5093 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 13:03:00.912881 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.907995    5093 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 13:03:00.912881 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.907999    5093 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 13:03:00.912881 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908003    5093 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 13:03:00.912881 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908009    5093 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 13:03:00.913442 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908013    5093 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 13:03:00.913442 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908018    5093 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 13:03:00.913442 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908022    5093 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 13:03:00.913442 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908027    5093 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 13:03:00.913442 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908031    5093 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 13:03:00.913442 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908036    5093 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 13:03:00.913442 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908042    5093 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 13:03:00.913442 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908047    5093 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 13:03:00.913442 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908051    5093 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 13:03:00.913442 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908055    5093 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 13:03:00.913442 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908059    5093 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 13:03:00.913442 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908064    5093 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 13:03:00.913442 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908068    5093 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 13:03:00.913442 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908072    5093 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 13:03:00.913442 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908077    5093 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 13:03:00.913442 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908081    5093 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 13:03:00.913442 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908085    5093 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 13:03:00.913442 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908090    5093 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 13:03:00.913442 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908095    5093 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 13:03:00.913442 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908099    5093 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 13:03:00.914189 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908103    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 13:03:00.914189 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908107    5093 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 13:03:00.914189 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908111    5093 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 13:03:00.914189 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908116    5093 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 13:03:00.914189 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908120    5093 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 13:03:00.914189 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908142    5093 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 13:03:00.914189 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908147    5093 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 13:03:00.914189 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908152    5093 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 13:03:00.914189 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908156    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 13:03:00.914189 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908160    5093 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 13:03:00.914189 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908165    5093 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 13:03:00.914189 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908168    5093 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 13:03:00.914189 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908173    5093 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 13:03:00.914189 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908177    5093 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 13:03:00.914189 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908181    5093 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 13:03:00.914189 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908186    5093 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 13:03:00.914189 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908190    5093 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 13:03:00.914189 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908197    5093 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 13:03:00.914189 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908206    5093 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 13:03:00.914822 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908211    5093 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 13:03:00.914822 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908216    5093 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 13:03:00.914822 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908221    5093 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 13:03:00.914822 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908225    5093 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 13:03:00.914822 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908230    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 13:03:00.914822 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908234    5093 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 13:03:00.914822 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908239    5093 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 13:03:00.914822 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908244    5093 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 13:03:00.914822 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908248    5093 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 13:03:00.914822 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908253    5093 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 13:03:00.914822 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908257    5093 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 13:03:00.914822 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908261    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 13:03:00.914822 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908266    5093 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 13:03:00.914822 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908270    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 13:03:00.914822 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908275    5093 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 13:03:00.914822 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908280    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 13:03:00.914822 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908284    5093 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 13:03:00.914822 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908289    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 13:03:00.914822 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908295    5093 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 13:03:00.915309 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908301    5093 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 13:03:00.915309 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908305    5093 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 13:03:00.915309 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908310    5093 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 13:03:00.915309 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908314    5093 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 13:03:00.915309 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908318    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 13:03:00.915309 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908323    5093 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 13:03:00.915309 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908327    5093 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 13:03:00.915309 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908332    5093 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 13:03:00.915309 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908336    5093 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 13:03:00.915309 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908340    5093 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 13:03:00.915309 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908344    5093 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 13:03:00.915309 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908349    5093 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 13:03:00.915309 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908354    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 13:03:00.915309 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908359    5093 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 13:03:00.915309 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908364    5093 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 13:03:00.915309 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908368    5093 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 13:03:00.915309 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908372    5093 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 13:03:00.915309 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908377    5093 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 13:03:00.915309 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908381    5093 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 13:03:00.915772 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908385    5093 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 13:03:00.915772 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908389    5093 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 13:03:00.915772 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908393    5093 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 13:03:00.915772 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908398    5093 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 13:03:00.915772 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908402    5093 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 13:03:00.915772 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908406    5093 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 13:03:00.915772 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908411    5093 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 13:03:00.915772 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908415    5093 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 13:03:00.915772 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908419    5093 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 13:03:00.915772 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908423    5093 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 13:03:00.915772 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908428    5093 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 13:03:00.915772 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908432    5093 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 13:03:00.915772 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908436    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 13:03:00.915772 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908441    5093 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 13:03:00.915772 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908445    5093 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 13:03:00.915772 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908449    5093 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 13:03:00.915772 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908453    5093 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 13:03:00.915772 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908457    5093 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 13:03:00.915772 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908462    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 13:03:00.915772 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908466    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 13:03:00.916275 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908470    5093 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 13:03:00.916275 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.908476    5093 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 13:03:00.916275 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908617    5093 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 13:03:00.916275 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908628    5093 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 13:03:00.916275 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908643    5093 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 13:03:00.916275 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908651    5093 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 13:03:00.916275 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908658    5093 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 13:03:00.916275 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908664    5093 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 13:03:00.916275 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908672    5093 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 13:03:00.916275 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908679    5093 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 13:03:00.916275 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908684    5093 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 13:03:00.916275 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908689    5093 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 13:03:00.916275 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908694    5093 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 13:03:00.916275 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908699    5093 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 13:03:00.916275 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908705    5093 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 13:03:00.916275 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908710    5093 flags.go:64] FLAG: --cgroup-root=""
Oct 30 13:03:00.916275 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908714    5093 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 13:03:00.916275 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908719    5093 flags.go:64] FLAG: --client-ca-file=""
Oct 30 13:03:00.916275 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908724    5093 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 13:03:00.916275 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908729    5093 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 13:03:00.916275 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908736    5093 flags.go:64] FLAG: --cluster-domain=""
Oct 30 13:03:00.916275 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908741    5093 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 13:03:00.916275 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908746    5093 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 13:03:00.916275 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908751    5093 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 13:03:00.916835 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908757    5093 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 13:03:00.916835 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908764    5093 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 13:03:00.916835 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908769    5093 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 13:03:00.916835 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908774    5093 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 13:03:00.916835 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908779    5093 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 13:03:00.916835 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908784    5093 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 13:03:00.916835 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908788    5093 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 13:03:00.916835 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908793    5093 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 13:03:00.916835 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908798    5093 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 13:03:00.916835 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908803    5093 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 13:03:00.916835 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908811    5093 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 13:03:00.916835 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908816    5093 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 13:03:00.916835 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908823    5093 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 13:03:00.916835 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908828    5093 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 13:03:00.916835 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908833    5093 flags.go:64] FLAG: --enable-server="true"
Oct 30 13:03:00.916835 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908838    5093 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 13:03:00.916835 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908845    5093 flags.go:64] FLAG: --event-burst="100"
Oct 30 13:03:00.916835 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908850    5093 flags.go:64] FLAG: --event-qps="50"
Oct 30 13:03:00.916835 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908855    5093 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 13:03:00.916835 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908860    5093 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 13:03:00.916835 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908864    5093 flags.go:64] FLAG: --eviction-hard=""
Oct 30 13:03:00.916835 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908873    5093 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 13:03:00.916835 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908877    5093 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 13:03:00.916835 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908883    5093 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 13:03:00.916835 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908888    5093 flags.go:64] FLAG: --eviction-soft=""
Oct 30 13:03:00.917484 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908894    5093 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 13:03:00.917484 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908899    5093 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 13:03:00.917484 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908904    5093 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 13:03:00.917484 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908909    5093 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 13:03:00.917484 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908913    5093 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 13:03:00.917484 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908918    5093 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 13:03:00.917484 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908923    5093 flags.go:64] FLAG: --feature-gates=""
Oct 30 13:03:00.917484 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908930    5093 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 13:03:00.917484 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908935    5093 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 13:03:00.917484 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908940    5093 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 13:03:00.917484 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908945    5093 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 13:03:00.917484 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908950    5093 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 13:03:00.917484 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908955    5093 flags.go:64] FLAG: --help="false"
Oct 30 13:03:00.917484 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908959    5093 flags.go:64] FLAG: --hostname-override="ip-10-0-134-31.ec2.internal"
Oct 30 13:03:00.917484 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908965    5093 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 13:03:00.917484 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908970    5093 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 13:03:00.917484 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908974    5093 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 13:03:00.917484 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908980    5093 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 13:03:00.917484 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908986    5093 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 13:03:00.917484 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908993    5093 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 13:03:00.917484 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.908998    5093 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 13:03:00.917484 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909002    5093 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 13:03:00.917484 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909007    5093 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 13:03:00.918044 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909012    5093 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 13:03:00.918044 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909020    5093 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 13:03:00.918044 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909025    5093 flags.go:64] FLAG: --kube-reserved=""
Oct 30 13:03:00.918044 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909030    5093 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 13:03:00.918044 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909034    5093 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 13:03:00.918044 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909039    5093 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 13:03:00.918044 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909044    5093 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 13:03:00.918044 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909048    5093 flags.go:64] FLAG: --lock-file=""
Oct 30 13:03:00.918044 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909053    5093 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 13:03:00.918044 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909058    5093 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 13:03:00.918044 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909063    5093 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 13:03:00.918044 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909072    5093 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 13:03:00.918044 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909079    5093 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 13:03:00.918044 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909084    5093 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 13:03:00.918044 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909088    5093 flags.go:64] FLAG: --logging-format="text"
Oct 30 13:03:00.918044 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909093    5093 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 13:03:00.918044 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909098    5093 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 13:03:00.918044 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909103    5093 flags.go:64] FLAG: --manifest-url=""
Oct 30 13:03:00.918044 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909108    5093 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 13:03:00.918044 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909115    5093 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 13:03:00.918044 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909121    5093 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 13:03:00.918044 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909146    5093 flags.go:64] FLAG: --max-pods="110"
Oct 30 13:03:00.918044 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909152    5093 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 13:03:00.918044 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909157    5093 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 13:03:00.918044 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909161    5093 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 13:03:00.918692 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909166    5093 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 13:03:00.918692 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909172    5093 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 13:03:00.918692 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909176    5093 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 13:03:00.918692 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909183    5093 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 13:03:00.918692 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909194    5093 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 13:03:00.918692 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909199    5093 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 13:03:00.918692 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909205    5093 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 13:03:00.918692 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909210    5093 flags.go:64] FLAG: --pod-cidr=""
Oct 30 13:03:00.918692 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909214    5093 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 13:03:00.918692 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909220    5093 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 13:03:00.918692 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909225    5093 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 13:03:00.918692 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909230    5093 flags.go:64] FLAG: --port="10250"
Oct 30 13:03:00.918692 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909235    5093 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 13:03:00.918692 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909239    5093 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c1fca1c339c24a35"
Oct 30 13:03:00.918692 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909245    5093 flags.go:64] FLAG: --qos-reserved=""
Oct 30 13:03:00.918692 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909250    5093 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 13:03:00.918692 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909255    5093 flags.go:64] FLAG: --register-node="true"
Oct 30 13:03:00.918692 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909260    5093 flags.go:64] FLAG: --register-with-taints=""
Oct 30 13:03:00.918692 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909267    5093 flags.go:64] FLAG: --registry-burst="10"
Oct 30 13:03:00.918692 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909273    5093 flags.go:64] FLAG: --registry-qps="5"
Oct 30 13:03:00.918692 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909278    5093 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 13:03:00.918692 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909283    5093 flags.go:64] FLAG: --reserved-memory=""
Oct 30 13:03:00.918692 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909290    5093 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 13:03:00.918692 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909295    5093 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 13:03:00.918692 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909300    5093 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 13:03:00.918692 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909305    5093 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 13:03:00.919329 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909309    5093 flags.go:64] FLAG: --runonce="false"
Oct 30 13:03:00.919329 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909313    5093 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 13:03:00.919329 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909318    5093 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 13:03:00.919329 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909323    5093 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 13:03:00.919329 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909329    5093 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 13:03:00.919329 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909335    5093 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 13:03:00.919329 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909340    5093 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 13:03:00.919329 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909345    5093 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 13:03:00.919329 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909350    5093 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 13:03:00.919329 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909357    5093 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 13:03:00.919329 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909361    5093 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 13:03:00.919329 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909366    5093 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 13:03:00.919329 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909372    5093 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 13:03:00.919329 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909377    5093 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 13:03:00.919329 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909382    5093 flags.go:64] FLAG: --system-cgroups=""
Oct 30 13:03:00.919329 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909387    5093 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 13:03:00.919329 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909397    5093 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 13:03:00.919329 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909402    5093 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 13:03:00.919329 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909407    5093 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 13:03:00.919329 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909421    5093 flags.go:64] FLAG: --tls-min-version=""
Oct 30 13:03:00.919329 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909426    5093 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 13:03:00.919329 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909431    5093 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 13:03:00.919329 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909437    5093 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 13:03:00.919329 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909442    5093 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 13:03:00.919329 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909447    5093 flags.go:64] FLAG: --v="2"
Oct 30 13:03:00.919923 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909456    5093 flags.go:64] FLAG: --version="false"
Oct 30 13:03:00.919923 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909463    5093 flags.go:64] FLAG: --vmodule=""
Oct 30 13:03:00.919923 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909469    5093 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 13:03:00.919923 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.909475    5093 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 13:03:00.919923 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909699    5093 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 13:03:00.919923 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909706    5093 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 13:03:00.919923 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909710    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 13:03:00.919923 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909715    5093 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 13:03:00.919923 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909719    5093 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 13:03:00.919923 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909724    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 13:03:00.919923 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909728    5093 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 13:03:00.919923 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909732    5093 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 13:03:00.919923 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909738    5093 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 13:03:00.919923 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909742    5093 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 13:03:00.919923 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909746    5093 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 13:03:00.919923 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909751    5093 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 13:03:00.919923 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909757    5093 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 13:03:00.919923 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909762    5093 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 13:03:00.919923 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909766    5093 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 13:03:00.919923 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909772    5093 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 13:03:00.920432 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909778    5093 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 13:03:00.920432 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909783    5093 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 13:03:00.920432 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909787    5093 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 13:03:00.920432 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909792    5093 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 13:03:00.920432 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909796    5093 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 13:03:00.920432 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909801    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 13:03:00.920432 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909806    5093 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 13:03:00.920432 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909810    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 13:03:00.920432 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909814    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 13:03:00.920432 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909819    5093 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 13:03:00.920432 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909824    5093 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 13:03:00.920432 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909828    5093 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 13:03:00.920432 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909834    5093 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 13:03:00.920432 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909838    5093 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 13:03:00.920432 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909842    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 13:03:00.920432 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909847    5093 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 13:03:00.920432 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909851    5093 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 13:03:00.920432 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909855    5093 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 13:03:00.920432 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909860    5093 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 13:03:00.920937 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909864    5093 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 13:03:00.920937 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909869    5093 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 13:03:00.920937 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909873    5093 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 13:03:00.920937 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909877    5093 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 13:03:00.920937 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909881    5093 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 13:03:00.920937 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909886    5093 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 13:03:00.920937 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909890    5093 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 13:03:00.920937 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909894    5093 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 13:03:00.920937 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909899    5093 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 13:03:00.920937 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909905    5093 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 13:03:00.920937 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909910    5093 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 13:03:00.920937 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909914    5093 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 13:03:00.920937 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909918    5093 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 13:03:00.920937 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909922    5093 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 13:03:00.920937 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909927    5093 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 13:03:00.920937 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909931    5093 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 13:03:00.920937 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909935    5093 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 13:03:00.920937 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909947    5093 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 13:03:00.920937 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909952    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 13:03:00.921566 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909957    5093 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 13:03:00.921566 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909961    5093 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 13:03:00.921566 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909965    5093 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 13:03:00.921566 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909970    5093 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 13:03:00.921566 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909974    5093 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 13:03:00.921566 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909978    5093 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 13:03:00.921566 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909984    5093 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 13:03:00.921566 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909988    5093 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 13:03:00.921566 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909993    5093 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 13:03:00.921566 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.909997    5093 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 13:03:00.921566 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910001    5093 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 13:03:00.921566 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910008    5093 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 13:03:00.921566 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910015    5093 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 13:03:00.921566 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910021    5093 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 13:03:00.921566 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910025    5093 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 13:03:00.921566 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910030    5093 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 13:03:00.921566 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910035    5093 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 13:03:00.921566 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910040    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 13:03:00.921566 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910044    5093 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 13:03:00.922081 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910048    5093 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 13:03:00.922081 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910052    5093 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 13:03:00.922081 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910057    5093 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 13:03:00.922081 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910063    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 13:03:00.922081 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910067    5093 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 13:03:00.922081 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910071    5093 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 13:03:00.922081 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910075    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 13:03:00.922081 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910080    5093 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 13:03:00.922081 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910084    5093 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 13:03:00.922081 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910089    5093 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 13:03:00.922081 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910093    5093 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 13:03:00.922081 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910097    5093 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 13:03:00.922081 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910101    5093 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 13:03:00.922081 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910106    5093 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 13:03:00.922081 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910111    5093 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 13:03:00.922081 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910115    5093 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 13:03:00.922081 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910119    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 13:03:00.922081 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910123    5093 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 13:03:00.922081 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910143    5093 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 13:03:00.922081 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910150    5093 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 13:03:00.922666 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910155    5093 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 13:03:00.922666 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910159    5093 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 13:03:00.922666 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910164    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 13:03:00.922666 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910168    5093 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 13:03:00.922666 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910173    5093 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 13:03:00.922666 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910177    5093 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 13:03:00.922666 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910181    5093 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 13:03:00.922666 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910185    5093 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 13:03:00.922666 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910189    5093 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 13:03:00.922666 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910194    5093 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 13:03:00.922666 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910199    5093 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 13:03:00.922666 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910203    5093 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 13:03:00.922666 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910208    5093 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 13:03:00.922666 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.910256    5093 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 13:03:00.922666 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910315    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 13:03:00.922666 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910324    5093 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 13:03:00.923059 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910329    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 13:03:00.923059 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910334    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 13:03:00.923059 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910339    5093 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 13:03:00.923059 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910343    5093 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 13:03:00.923059 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910348    5093 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 13:03:00.923059 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910352    5093 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 13:03:00.923059 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910357    5093 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 13:03:00.923059 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910361    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 13:03:00.923059 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910366    5093 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 13:03:00.923059 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910370    5093 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 13:03:00.923059 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910375    5093 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 13:03:00.923059 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910380    5093 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 13:03:00.923059 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910384    5093 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 13:03:00.923059 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910388    5093 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 13:03:00.923059 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910393    5093 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 13:03:00.923059 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910397    5093 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 13:03:00.923059 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910404    5093 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 13:03:00.923059 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910408    5093 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 13:03:00.923059 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910412    5093 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 13:03:00.923530 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910416    5093 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 13:03:00.923530 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910421    5093 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 13:03:00.923530 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910425    5093 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 13:03:00.923530 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910430    5093 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 13:03:00.923530 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910434    5093 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 13:03:00.923530 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910438    5093 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 13:03:00.923530 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910443    5093 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 13:03:00.923530 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910447    5093 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 13:03:00.923530 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910451    5093 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 13:03:00.923530 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910455    5093 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 13:03:00.923530 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910460    5093 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 13:03:00.923530 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910464    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 13:03:00.923530 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910470    5093 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 13:03:00.923530 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910474    5093 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 13:03:00.923530 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910478    5093 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 13:03:00.923530 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910482    5093 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 13:03:00.923530 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910486    5093 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 13:03:00.923530 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910490    5093 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 13:03:00.923530 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910494    5093 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 13:03:00.923530 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910498    5093 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 13:03:00.924268 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910503    5093 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 13:03:00.924268 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910507    5093 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 13:03:00.924268 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910511    5093 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 13:03:00.924268 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910519    5093 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 13:03:00.924268 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910526    5093 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 13:03:00.924268 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910531    5093 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 13:03:00.924268 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910536    5093 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 13:03:00.924268 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910541    5093 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 13:03:00.924268 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910545    5093 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 13:03:00.924268 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910550    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 13:03:00.924268 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910554    5093 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 13:03:00.924268 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910559    5093 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 13:03:00.924268 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910563    5093 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 13:03:00.924268 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910567    5093 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 13:03:00.924268 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910572    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 13:03:00.924268 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910576    5093 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 13:03:00.924268 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910580    5093 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 13:03:00.924268 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910584    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 13:03:00.924268 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910588    5093 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 13:03:00.924795 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910592    5093 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 13:03:00.924795 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910597    5093 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 13:03:00.924795 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910601    5093 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 13:03:00.924795 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910606    5093 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 13:03:00.924795 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910610    5093 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 13:03:00.924795 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910617    5093 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 13:03:00.924795 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910621    5093 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 13:03:00.924795 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910625    5093 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 13:03:00.924795 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910629    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 13:03:00.924795 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910634    5093 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 13:03:00.924795 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910638    5093 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 13:03:00.924795 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910642    5093 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 13:03:00.924795 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910646    5093 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 13:03:00.924795 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910651    5093 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 13:03:00.924795 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910655    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 13:03:00.924795 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910659    5093 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 13:03:00.924795 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910664    5093 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 13:03:00.924795 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910668    5093 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 13:03:00.924795 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910673    5093 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 13:03:00.924795 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910677    5093 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 13:03:00.925399 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910681    5093 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 13:03:00.925399 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910686    5093 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 13:03:00.925399 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910691    5093 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 13:03:00.925399 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910695    5093 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 13:03:00.925399 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910699    5093 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 13:03:00.925399 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910704    5093 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 13:03:00.925399 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910708    5093 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 13:03:00.925399 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910712    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 13:03:00.925399 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910716    5093 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 13:03:00.925399 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910720    5093 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 13:03:00.925399 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910725    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 13:03:00.925399 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910729    5093 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 13:03:00.925399 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910733    5093 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 13:03:00.925399 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910738    5093 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 13:03:00.925399 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910742    5093 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 13:03:00.925399 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910746    5093 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 13:03:00.925399 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910750    5093 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 13:03:00.925399 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910762    5093 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 13:03:00.925399 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910767    5093 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 13:03:00.925399 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910771    5093 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 13:03:00.925871 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910780    5093 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 13:03:00.925871 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910786    5093 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 13:03:00.925871 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910790    5093 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 13:03:00.925871 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910795    5093 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 13:03:00.925871 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910799    5093 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 13:03:00.925871 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.910803    5093 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 13:03:00.925871 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.916340    5093 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 13:03:00.925871 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.916358    5093 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 13:03:00.925871 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.916377    5093 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 13:03:00.925871 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.916382    5093 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 13:03:00.925871 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916442    5093 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 13:03:00.925871 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916448    5093 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 13:03:00.925871 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916451    5093 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 13:03:00.925871 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916454    5093 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 13:03:00.925871 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916457    5093 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 13:03:00.925871 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916460    5093 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 13:03:00.925871 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916463    5093 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 13:03:00.925871 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916466    5093 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 13:03:00.925871 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916470    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 13:03:00.926361 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916473    5093 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 13:03:00.926361 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916475    5093 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 13:03:00.926361 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916478    5093 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 13:03:00.926361 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916481    5093 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 13:03:00.926361 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916483    5093 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 13:03:00.926361 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916493    5093 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 13:03:00.926361 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916497    5093 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 13:03:00.926361 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916500    5093 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 13:03:00.926361 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916503    5093 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 13:03:00.926361 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916506    5093 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 13:03:00.926361 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916508    5093 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 13:03:00.926361 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916511    5093 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 13:03:00.926361 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916514    5093 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 13:03:00.926361 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916516    5093 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 13:03:00.926361 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916519    5093 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 13:03:00.926361 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916521    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 13:03:00.926361 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916524    5093 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 13:03:00.926361 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916526    5093 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 13:03:00.926361 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916530    5093 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 13:03:00.926361 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916533    5093 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 13:03:00.926838 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916535    5093 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 13:03:00.926838 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916538    5093 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 13:03:00.926838 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916541    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 13:03:00.926838 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916544    5093 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 13:03:00.926838 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916546    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 13:03:00.926838 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916549    5093 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 13:03:00.926838 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916551    5093 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 13:03:00.926838 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916554    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 13:03:00.926838 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916557    5093 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 13:03:00.926838 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916560    5093 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 13:03:00.926838 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916562    5093 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 13:03:00.926838 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916565    5093 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 13:03:00.926838 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916569    5093 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 13:03:00.926838 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916574    5093 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 13:03:00.926838 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916577    5093 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 13:03:00.926838 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916580    5093 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 13:03:00.926838 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916582    5093 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 13:03:00.926838 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916585    5093 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 13:03:00.926838 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916588    5093 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 13:03:00.927323 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916591    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 13:03:00.927323 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916593    5093 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 13:03:00.927323 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916596    5093 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 13:03:00.927323 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916598    5093 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 13:03:00.927323 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916601    5093 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 13:03:00.927323 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916604    5093 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 13:03:00.927323 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916606    5093 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 13:03:00.927323 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916609    5093 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 13:03:00.927323 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916612    5093 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 13:03:00.927323 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916614    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 13:03:00.927323 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916617    5093 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 13:03:00.927323 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916620    5093 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 13:03:00.927323 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916623    5093 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 13:03:00.927323 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916626    5093 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 13:03:00.927323 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916628    5093 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 13:03:00.927323 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916631    5093 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 13:03:00.927323 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916633    5093 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 13:03:00.927323 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916636    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 13:03:00.927323 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916639    5093 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 13:03:00.927323 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916641    5093 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 13:03:00.927793 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916644    5093 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 13:03:00.927793 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916647    5093 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 13:03:00.927793 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916649    5093 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 13:03:00.927793 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916652    5093 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 13:03:00.927793 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916655    5093 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 13:03:00.927793 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916657    5093 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 13:03:00.927793 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916660    5093 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 13:03:00.927793 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916662    5093 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 13:03:00.927793 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916665    5093 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 13:03:00.927793 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916667    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 13:03:00.927793 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916670    5093 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 13:03:00.927793 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916673    5093 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 13:03:00.927793 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916676    5093 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 13:03:00.927793 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916678    5093 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 13:03:00.927793 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916682    5093 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 13:03:00.927793 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916685    5093 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 13:03:00.927793 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916689    5093 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 13:03:00.927793 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916692    5093 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 13:03:00.927793 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916695    5093 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 13:03:00.928260 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916699    5093 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 13:03:00.928260 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916701    5093 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 13:03:00.928260 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916704    5093 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 13:03:00.928260 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916707    5093 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 13:03:00.928260 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916710    5093 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 13:03:00.928260 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916713    5093 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 13:03:00.928260 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916715    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 13:03:00.928260 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916718    5093 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 13:03:00.928260 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916720    5093 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 13:03:00.928260 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916723    5093 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 13:03:00.928260 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916726    5093 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 13:03:00.928260 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916728    5093 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 13:03:00.928260 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916731    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 13:03:00.928260 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916733    5093 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 13:03:00.928260 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916736    5093 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 13:03:00.928260 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916738    5093 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 13:03:00.928260 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916741    5093 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 13:03:00.928260 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916743    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 13:03:00.928260 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916746    5093 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 13:03:00.928715 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.916775    5093 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 13:03:00.928715 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916922    5093 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 13:03:00.928715 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916926    5093 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 13:03:00.928715 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916929    5093 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 13:03:00.928715 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916932    5093 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 13:03:00.928715 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916934    5093 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 13:03:00.928715 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916937    5093 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 13:03:00.928715 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916940    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 13:03:00.928715 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916943    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 13:03:00.928715 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916945    5093 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 13:03:00.928715 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916948    5093 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 13:03:00.928715 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916951    5093 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 13:03:00.928715 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916953    5093 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 13:03:00.928715 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916956    5093 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 13:03:00.928715 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916959    5093 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 13:03:00.928715 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916961    5093 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 13:03:00.929099 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916964    5093 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 13:03:00.929099 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916967    5093 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 13:03:00.929099 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916969    5093 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 13:03:00.929099 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916972    5093 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 13:03:00.929099 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916975    5093 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 13:03:00.929099 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916977    5093 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 13:03:00.929099 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916980    5093 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 13:03:00.929099 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916982    5093 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 13:03:00.929099 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916985    5093 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 13:03:00.929099 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916988    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 13:03:00.929099 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916992    5093 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 13:03:00.929099 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.916997    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 13:03:00.929099 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917001    5093 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 13:03:00.929099 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917006    5093 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 13:03:00.929099 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917009    5093 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 13:03:00.929099 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917012    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 13:03:00.929099 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917025    5093 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 13:03:00.929099 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917028    5093 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 13:03:00.929099 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917030    5093 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 13:03:00.929099 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917033    5093 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 13:03:00.929811 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917036    5093 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 13:03:00.929811 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917038    5093 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 13:03:00.929811 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917041    5093 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 13:03:00.929811 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917046    5093 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 13:03:00.929811 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917048    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 13:03:00.929811 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917051    5093 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 13:03:00.929811 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917054    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 13:03:00.929811 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917057    5093 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 13:03:00.929811 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917059    5093 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 13:03:00.929811 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917062    5093 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 13:03:00.929811 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917065    5093 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 13:03:00.929811 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917068    5093 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 13:03:00.929811 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917070    5093 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 13:03:00.929811 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917073    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 13:03:00.929811 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917076    5093 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 13:03:00.929811 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917079    5093 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 13:03:00.929811 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917081    5093 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 13:03:00.929811 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917084    5093 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 13:03:00.929811 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917086    5093 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 13:03:00.929811 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917089    5093 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 13:03:00.930349 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917091    5093 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 13:03:00.930349 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917094    5093 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 13:03:00.930349 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917096    5093 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 13:03:00.930349 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917099    5093 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 13:03:00.930349 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917104    5093 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 13:03:00.930349 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917108    5093 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 13:03:00.930349 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917111    5093 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 13:03:00.930349 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917114    5093 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 13:03:00.930349 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917116    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 13:03:00.930349 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917119    5093 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 13:03:00.930349 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917121    5093 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 13:03:00.930349 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917124    5093 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 13:03:00.930349 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917146    5093 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 13:03:00.930349 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917149    5093 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 13:03:00.930349 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917151    5093 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 13:03:00.930349 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917154    5093 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 13:03:00.930349 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917157    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 13:03:00.930349 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917160    5093 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 13:03:00.930349 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917163    5093 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 13:03:00.930349 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917165    5093 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 13:03:00.930827 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917168    5093 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 13:03:00.930827 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917170    5093 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 13:03:00.930827 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917173    5093 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 13:03:00.930827 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917176    5093 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 13:03:00.930827 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917179    5093 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 13:03:00.930827 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917181    5093 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 13:03:00.930827 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917184    5093 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 13:03:00.930827 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917187    5093 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 13:03:00.930827 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917190    5093 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 13:03:00.930827 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917192    5093 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 13:03:00.930827 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917195    5093 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 13:03:00.930827 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917197    5093 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 13:03:00.930827 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917200    5093 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 13:03:00.930827 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917202    5093 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 13:03:00.930827 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917207    5093 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 13:03:00.930827 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917211    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 13:03:00.930827 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917214    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 13:03:00.930827 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917216    5093 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 13:03:00.931277 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917219    5093 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 13:03:00.931277 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917221    5093 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 13:03:00.931277 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917224    5093 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 13:03:00.931277 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917227    5093 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 13:03:00.931277 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917230    5093 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 13:03:00.931277 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917232    5093 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 13:03:00.931277 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917235    5093 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 13:03:00.931277 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917238    5093 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 13:03:00.931277 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917240    5093 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 13:03:00.931277 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917243    5093 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 13:03:00.931277 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917245    5093 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 13:03:00.931277 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917248    5093 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 13:03:00.931277 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917251    5093 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 13:03:00.931277 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.917284    5093 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 13:03:00.931277 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917320    5093 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 13:03:00.931277 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917324    5093 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 13:03:00.931696 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917327    5093 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 13:03:00.931696 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917330    5093 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 13:03:00.931696 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917333    5093 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 13:03:00.931696 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917336    5093 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 13:03:00.931696 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917338    5093 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 13:03:00.931696 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917341    5093 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 13:03:00.931696 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917343    5093 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 13:03:00.931696 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917346    5093 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 13:03:00.931696 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917348    5093 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 13:03:00.931696 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917351    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 13:03:00.931696 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917353    5093 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 13:03:00.931696 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917356    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 13:03:00.931696 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917358    5093 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 13:03:00.931696 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917361    5093 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 13:03:00.931696 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917364    5093 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 13:03:00.931696 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917366    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 13:03:00.931696 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917369    5093 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 13:03:00.931696 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917371    5093 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 13:03:00.931696 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917374    5093 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 13:03:00.931696 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917377    5093 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 13:03:00.932340 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917379    5093 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 13:03:00.932340 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917382    5093 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 13:03:00.932340 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917384    5093 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 13:03:00.932340 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917387    5093 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 13:03:00.932340 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917389    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 13:03:00.932340 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917392    5093 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 13:03:00.932340 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917394    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 13:03:00.932340 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917397    5093 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 13:03:00.932340 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917400    5093 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 13:03:00.932340 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917406    5093 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 13:03:00.932340 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917409    5093 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 13:03:00.932340 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917411    5093 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 13:03:00.932340 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917414    5093 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 13:03:00.932340 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917416    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 13:03:00.932340 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917419    5093 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 13:03:00.932340 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917421    5093 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 13:03:00.932340 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917424    5093 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 13:03:00.932340 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917426    5093 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 13:03:00.932340 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917429    5093 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 13:03:00.932340 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917431    5093 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 13:03:00.933153 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917436    5093 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 13:03:00.933153 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917439    5093 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 13:03:00.933153 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917442    5093 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 13:03:00.933153 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917444    5093 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 13:03:00.933153 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917448    5093 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 13:03:00.933153 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917453    5093 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 13:03:00.933153 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917456    5093 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 13:03:00.933153 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917459    5093 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 13:03:00.933153 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917461    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 13:03:00.933153 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917464    5093 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 13:03:00.933153 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917467    5093 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 13:03:00.933153 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917469    5093 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 13:03:00.933153 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917472    5093 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 13:03:00.933153 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917474    5093 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 13:03:00.933153 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917477    5093 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 13:03:00.933153 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917480    5093 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 13:03:00.933153 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917483    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 13:03:00.933153 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917485    5093 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 13:03:00.933153 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917488    5093 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 13:03:00.933153 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917491    5093 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 13:03:00.933749 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917493    5093 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 13:03:00.933749 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917499    5093 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 13:03:00.933749 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917503    5093 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 13:03:00.933749 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917505    5093 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 13:03:00.933749 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917508    5093 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 13:03:00.933749 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917511    5093 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 13:03:00.933749 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917513    5093 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 13:03:00.933749 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917516    5093 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 13:03:00.933749 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917519    5093 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 13:03:00.933749 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917521    5093 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 13:03:00.933749 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917524    5093 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 13:03:00.933749 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917527    5093 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 13:03:00.933749 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917531    5093 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 13:03:00.933749 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917534    5093 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 13:03:00.933749 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917538    5093 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 13:03:00.933749 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917542    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 13:03:00.933749 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917545    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 13:03:00.933749 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917548    5093 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 13:03:00.934323 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917551    5093 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 13:03:00.934323 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917554    5093 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 13:03:00.934323 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917556    5093 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 13:03:00.934323 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917559    5093 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 13:03:00.934323 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917562    5093 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 13:03:00.934323 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917565    5093 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 13:03:00.934323 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917568    5093 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 13:03:00.934323 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917571    5093 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 13:03:00.934323 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917573    5093 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 13:03:00.934323 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917576    5093 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 13:03:00.934323 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917578    5093 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 13:03:00.934323 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917581    5093 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 13:03:00.934323 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917584    5093 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 13:03:00.934323 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917586    5093 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 13:03:00.934323 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917589    5093 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 13:03:00.934323 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917593    5093 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 13:03:00.934323 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917596    5093 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 13:03:00.934323 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917599    5093 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 13:03:00.934323 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917602    5093 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 13:03:00.934323 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917604    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 13:03:00.934960 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917607    5093 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 13:03:00.934960 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917609    5093 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 13:03:00.934960 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917612    5093 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 13:03:00.934960 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917614    5093 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 13:03:00.934960 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917617    5093 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 13:03:00.934960 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:00.917619    5093 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 13:03:00.934960 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.917680    5093 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 13:03:00.934960 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.918668    5093 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 13:03:00.934960 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.918742    5093 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 13:03:00.934960 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.918933    5093 server.go:1014] "Starting client certificate rotation"
Oct 30 13:03:00.934960 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.919074    5093 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 13:03:00.934960 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.919141    5093 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 12:57:19 +0000 UTC" deadline="2028-06-25 19:41:04.544324042 +0000 UTC"
Oct 30 13:03:00.934960 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.919182    5093 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14502h38m3.625144365s"
Oct 30 13:03:00.934960 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.919392    5093 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 13:03:00.934960 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.920511    5093 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 13:03:00.934960 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.921644    5093 log.go:25] "Validated CRI v1 runtime API"
Oct 30 13:03:00.935485 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.923336    5093 log.go:25] "Validated CRI v1 image API"
Oct 30 13:03:00.935485 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.924120    5093 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 13:03:00.935485 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.926247    5093 fs.go:135] Filesystem UUIDs: map[3162dc0a-d86f-4d0d-89f6-a49ea6a12806:/dev/nvme0n1p4 5336019c-c3a9-4bf3-a44b-982186316a05:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Oct 30 13:03:00.935622 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.926263    5093 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/09db080f67588acba375556f890373b7865b100a2a41cf985e15ec97b6c1899b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/09db080f67588acba375556f890373b7865b100a2a41cf985e15ec97b6c1899b/userdata/shm major:0 minor:342 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0aef0d1cf6af23a8dab50f6eb1d692bf62f8538e1dc9f793a6d8680e303e4732/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0aef0d1cf6af23a8dab50f6eb1d692bf62f8538e1dc9f793a6d8680e303e4732/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0b84f92815296a8a34a2754e0fa1c4510032d61e7fd9551ebf454424a49a192f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0b84f92815296a8a34a2754e0fa1c4510032d61e7fd9551ebf454424a49a192f/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0c41bc03f6369e26cb0fac4e724d818f33ed871dba80d4d83b87b5bd08b7c28f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0c41bc03f6369e26cb0fac4e724d818f33ed871dba80d4d83b87b5bd08b7c28f/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/12715619dec36263a869e0698ef7e31b9481effac5627057399505c660155ba4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/12715619dec36263a869e0698ef7e31b9481effac5627057399505c660155ba4/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1b27ed0977f245135de6ee6d9d6d381ba53831c5ba22d2699bab4c5db56f508c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1b27ed0977f245135de6ee6d9d6d381ba53831c5ba22d2699bab4c5db56f508c/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2e963d5ab87bb8f8412d3ed93ae6eef220d880e9b5a0350f3f0c8365b9b5d6b2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2e963d5ab87bb8f8412d3ed93ae6eef220d880e9b5a0350f3f0c8365b9b5d6b2/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2f036d26baa035a5f2f97f9c748b0997a23f90fc3a79944f898e4fc7ebf17f73/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2f036d26baa035a5f2f97f9c748b0997a23f90fc3a79944f898e4fc7ebf17f73/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3cc6fafce08f8f3d202026631327bb8e095f97ff186665b5b1996bd0fe1fbc6c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3cc6fafce08f8f3d202026631327bb8e095f97ff186665b5b1996bd0fe1fbc6c/userdata/shm major:0 minor:316 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3f90007edff3cc0f6929888fcff5cdf01af5714c69ea75da6d164524fa6c4ba6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3f90007edff3cc0f6929888fcff5cdf01af5714c69ea75da6d164524fa6c4ba6/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/46ff484b065035d2962579b1722fe86d2b4e9d2e4b5cea97bf770ae859b773bd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/46ff484b065035d2962579b1722fe86d2b4e9d2e4b5cea97bf770ae859b773bd/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4e3de8cc64e483689b44b655b9983ba4b90579eb7514936c00f38d8a8d37bed6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4e3de8cc64e483689b44b655b9983ba4b90579eb7514936c00f38d8a8d37bed6/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/562b94e335ef4ee49a4a2c8c6eea2920298d30c5531be03ac529afce7dc418e4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/562b94e335ef4ee49a4a2c8c6eea2920298d30c5531be03ac529afce7dc418e4/userdata/shm major:0 minor:219 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7569409b6e188b60e656069548c94067881169acb401a01f609ce63fa445e43c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7569409b6e188b60e656069548c94067881169acb401a01f609ce63fa445e43c/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9da5be1ca0326ab3d9989a5ac8b3610b3b4a0c14a8fb316621a2045016ce29a8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9da5be1ca0326ab3d9989a5ac8b3610b3b4a0c14a8fb316621a2045016ce29a8/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a911208b7a7070410d4ca2f43e084cbbbc08ab928ec7df2bb1a3524f92dc8607/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a911208b7a7070410d4ca2f43e084cbbbc08ab928ec7df2bb1a3524f92dc8607/userdata/shm major:0 minor:73 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a9872fe8d3163787d5f6fcf7d35449ec1edca5abd3d826b8108c491c819a3783/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a9872fe8d3163787d5f6fcf7d35449ec1edca5abd3d826b8108c491c819a3783/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c6bb98ae54b0111cebebb0ab298186b328e44754b62c8032028fd66414186c50/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c6bb98ae54b0111cebebb0ab298186b328e44754b62c8032028fd66414186c50/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d4c6b7cb6278106280fbe5bc2fb62290e5932aa7d28d851f97901a9af05cdb96/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d4c6b7cb6278106280fbe5bc2fb62290e5932aa7d28d851f97901a9af05cdb96/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/de331273dc14e36948f28a293bd31355eecd2f4777b203d2a0420a0f26b75b5f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/de331273dc14e36948f28a293bd31355eecd2f4777b203d2a0420a0f26b75b5f/userdata/shm major:0 minor:223 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dfe8cfde85a2c351da022ee8103df05f789b03b6ed99ffc5b984323d5cac5816/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dfe8cfde85a2c351da022ee8103df05f789b03b6ed99ffc5b984323d5cac5816/userdata/shm major:0 minor:358 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f1de5a011f5645942aa0dd822f5aae0ca5197e9b5b01aacafd9fa403fe7ed36d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f1de5a011f5645942aa0dd822f5aae0ca5197e9b5b01aacafd9fa403fe7ed36d/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f227bcba4afc53a24a6c3631b7e33b3cc84b82b155f54df82731be6b10bc97fc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f227bcba4afc53a24a6c3631b7e33b3cc84b82b155f54df82731be6b10bc97fc/userdata/shm major:0 minor:347 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f2aae6460a32ba64304b33d00d3f239e0ad112f520d74613426b5ca429211604/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f2aae6460a32ba64304b33d00d3f239e0ad112f520d74613426b5ca429211604/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f6950ee20e87263fb0dd7718cc81691d24936acf6bb32219f5659b8b23cfefdd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f6950ee20e87263fb0dd7718cc81691d24936acf6bb32219f5659b8b23cfefdd/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f816757e82dd09f5cfa0a0f30bfaf6c873fadd481a17dfde9e290f9900041ffb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f816757e82dd09f5cfa0a0f30bfaf6c873fadd481a17dfde9e290f9900041ffb/userdata/shm major:0 minor:221 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/087058f5-e286-4ac7-8ed8-a280da332878/volumes/kubernetes.io~projected/kube-api-access-6gdf9:{mountpoint:/var/lib/kubelet/pods/087058f5-e286-4ac7-8ed8-a280da332878/volumes/kubernetes.io~projected/kube-api-access-6gdf9 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c8e72ed-d54d-4e6a-bf66-65a16430351d/volumes/kubernetes.io~projected/kube-api-access-dgqhj:{mountpoint:/var/lib/kubelet/pods/0c8e72ed-d54d-4e6a-bf66-65a16430351d/volumes/kubernetes.io~projected/kube-api-access-dgqhj major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d0b2655-d52d-4c95-9589-d45237fa1057/volumes/kubernetes.io~projected/kube-api-access-plb6z:{mountpoint:/var/lib/kubelet/pods/0d0b2655-d52d-4c95-9589-d45237fa1057/volumes/kubernetes.io~projected/kube-api-access-plb6z major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10b774a1-1476-4f91-b6e5-d52b3f45700e/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/10b774a1-1476-4f91-b6e5-d52b3f45700e/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/155bbdd4-b566-467f-88c6-ec5b4b881a09/volumes/kubernetes.io~projected/kube-api-access-lsj7z:{mountpoint:/var/lib/kubelet/pods/155bbdd4-b566-467f-88c6-ec5b4b881a09/volumes/kubernetes.io~projected/kube-api-access-lsj7z major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/155bbdd4-b566-467f-88c6-ec5b4b881a09/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/155bbdd4-b566-467f-88c6-ec5b4b881a09/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/183f23e7-ae80-445b-960c-fff9b410ba5b/volumes/kubernetes.io~projected/kube-api-access-hjgw5:{mountpoint:/var/lib/kubelet/pods/183f23e7-ae80-445b-960c-fff9b410ba5b/volumes/kubernetes.io~projected/kube-api-access-hjgw5 major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a25d5a7-83d4-443b-98ae-fc7b603cfc5f/volumes/kubernetes.io~projected/kube-api-access-qk568:{mountpoint:/var/lib/kubelet/pods/1a25d5a7-83d4-443b-98ae-fc7b603cfc5f/volumes/kubernetes.io~projected/kube-api-access-qk568 major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a25d5a7-83d4-443b-98ae-fc7b603cfc5f/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/1a25d5a7-83d4-443b-98ae-fc7b603cfc5f/volumes/kubernetes.io~secret/cert major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a675ac7-fbe1-449b-accf-bb0456ed0bd3/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/3a675ac7-fbe1-449b-accf-bb0456ed0bd3/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a675ac7-fbe1-449b-accf-bb0456ed0bd3/volumes/kubernetes.io~projected/kube-api-access-xws2p:{mountpoint:/var/lib/kubelet/pods/3a675ac7-fbe1-449b-accf-bb0456ed0bd3/volumes/kubernetes.io~projected/kube-api-access-xws2p major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a675ac7-fbe1-449b-accf-bb0456ed0bd3/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/3a675ac7-fbe1-449b-accf-bb0456ed0bd3/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3c7ab011-8fd4-4d6d-b3f2-72d6af11d3e1/volumes/kubernetes.io~projected/kube-api-access-p4kjx:{mountpoint:/var/lib/kubelet/pods/3c7ab011-8fd4-4d6d-b3f2-72d6af11d3e1/volumes/kubernetes.io~projected/kube-api-access-p4kjx major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44f513ab-9115-4761-aa8b-c546212ec17b/volumes/kubernetes.io~projected/kube-api-access-vjjr5:{mountpoint:/var/lib/kubelet/pods/44f513ab-9115-4761-aa8b-c546212ec17b/volumes/kubernetes.io~projected/kube-api-access-vjjr5 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/46bedd0c-7237-4128-be98-69e4474bb535/volumes/kubernetes.io~projected/kube-api-access-kbpg6:{mountpoint:/var/lib/kubelet/pods/46bedd0c-7237-4128-be98-69e4474bb535/volumes/kubernetes.io~projected/kube-api-access-kbpg6 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ad66a0c-8dc0-40a4-a1b2-28d26b0de94b/volumes/kubernetes.io~projected/kube-api-access-rnx68:{mountpoint:/var/lib/kubelet/pods/4ad66a0c-8dc0-40a4-a1b2-28d26b0de94b/volumes/kubernetes.io~projected/kube-api-access-rnx68 major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a16e7b2-e45d-480a-a7bc-d914640b9867/volumes/kubernetes.io~projected/kube-api-access-z9m9z:{mountpoint:/var/lib/kubelet/pods/5a16e7b2-e45d-480a-a7bc-d914640b9867/volumes/kubernetes.io~projected/kube-api-access-z9m9z major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5cbda802-f819-4ea6-846e-440cea4afed2/volumes/kubernetes.io~projected/kube-api-access-48dsv:{mountpoint:/var/lib/kubelet/pods/5cbda802-f819-4ea6-846e-440cea4afed2/volumes/kubernetes.io~projected/kube-api-access-48dsv major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5cbda802-f819-4ea6-846e-440cea4afed2/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/5cbda802-f819-4ea6-846e-440cea4afed2/volumes/kubernetes.io~secret/serving-cert major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79664154-5454-4294-baad-12bdfad6fe68/volumes/kubernetes.io~projected/kube-api-access-7nrlb:{mountpoint:/var/lib/kubelet/pods/79664154-5454-4294-baad-12bdfad6fe68/volumes/kubernetes.io~projected/kube-api-access-7nrlb major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79664154-5454-4294-baad-12bdfad6fe68/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/79664154-5454-4294-baad-12bdfad6fe68/volumes/kubernetes.io~secret/serving-cert major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7bbbc7e0-1790-459b-952a-b6b7b4d55113/volumes/kubernetes.io~projected/kube-api-access-8jdqx:{mountpoint:/var/lib/kubelet/pods/7bbbc7e0-1790-459b-952a-b6b7b4d55113/volumes/kubernetes.io~projected/kube-api-access-8jdqx major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7cca8a39-bee0-4705-bcde-b31060e44944/volumes/kubernetes.io~projected/kube-api-access-kncj9:{mountpoint:/var/lib/kubelet/pods/7cca8a39-bee0-4705-bcde-b31060e44944/volumes/kubernetes.io~projected/kube-api-access-kncj9 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7ceb0bb1-def2-44a5-9e31-ceffd848125d/volumes/kubernetes.io~projected/kube-api-access-f5k78:{mountpoint:/var/lib/kubelet/pods/7ceb0bb1-def2-44a5-9e31-ceffd848125d/volumes/kubernetes.io~projected/kube-api-access-f5k78 major:0 minor:311 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d29458a-da6c-461c-9941-afa323061f30/volumes/kubernetes.io~projected/kube-api-access-789vz:{mountpoint:/var/lib/kubelet/pods/7d29458a-da6c-461c-9941-afa323061f30/volumes/kubernetes.io~projected/kube-api-access-789vz major:0 minor:355 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d29458a-da6c-461c-9941-afa323061f30/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/7d29458a-da6c-461c-9941-afa323061f30/volumes/kubernetes.io~secret/signing-key major:0 minor:354 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8021bc0b-4447-4f04-b9bd-5f80e9cf3507/volumes/kubernetes.io~projected/kube-api-access-262nq:{mountpoint:/var/lib/kubelet/pods/8021bc0b-4447-4f04-b9bd-5f80e9cf3507/volumes/kubernetes.io~projected/kube-api-access-262nq major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a07f5a09-920e-462f-ab6d-7a2ac53d199e/volumes/kubernetes.io~projected/kube-api-access-gz5ls:{mountpoint:/var/lib/kubelet/pods/a07f5a09-920e-462f-ab6d-7a2ac53d199e/volumes/kubernetes.io~projected/kube-api-access-gz5ls major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a323e312-dd91-486f-985d-92b9ab8cca04/volumes/kubernetes.io~projected/kube-api-access-whwnz:{mountpoint:/var/lib/kubelet/pods/a323e312-dd91-486f-985d-92b9ab8cca04/volumes/kubernetes.io~projected/kube-api-access-whwnz major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a32a253a-ecaf-4f90-af1c-27a7e53b9629/volumes/kubernetes.io~projected/kube-api-access-qsfql:{mountpoint:/var/lib/kubelet/pods/a32a253a-ecaf-4f90-af1c-27a7e53b9629/volumes/kubernetes.io~projected/kube-api-access-qsfql major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/abf08e3d-eec3-4f73-92a6-4ffb691cbd2d/volumes/kubernetes.io~projected/kube-api-access-tfxrl:{mountpoint:/var/lib/kubelet/pods/abf08e3d-eec3-4f73-92a6-4ffb691cbd2d/volumes/kubernetes.io~projected/kube-api-access-tfxrl major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5/volumes/kubernetes.io~projected/kube-api-access-vdrpm:{mountpoint:/var/lib/kubelet/pods/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5/volumes/kubernetes.io~projected/kube-api-access-vdrpm major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/beb2b7e4-b43d-42fc-9e8a-ecdfa6994205/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/beb2b7e4-b43d-42fc-9e8a-ecdfa6994205/volumes/kubernetes.io~secret/agent-certs major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5db3ba8-f1de-4fa5-9839-e7e8200da58f/volumes/kubernetes.io~projected/kube-api-access-rflhr:{mountpoint:/var/lib/kubelet/pods/d5db3ba8-f1de-4fa5-9839-e7e8200da58f/volumes/kubernetes.io~projected/kube-api-access-rflhr major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5db3ba8-f1de-4fa5-9839-e7e8200da58f/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/d5db3ba8-f1de-4fa5-9839-e7e8200da58f/volumes/kubernetes.io~secret/default-certificate major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5db3ba8-f1de-4fa5-9839-e7e8200da58f/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/d5db3ba8-f1de-4fa5-9839-e7e8200da58f/volumes/kubernetes.io~secret/stats-auth major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de69ec51-fd3a-4bc4-ac17-ff02bc582905/volumes/kubernetes.io~projected/kube-api-access-v6f7b:{mountpoint:/var/lib/kubelet/pods/de69ec51-fd3a-4bc4-ac17-ff02bc582905/volumes/kubernetes.io~projected/kube-api-access-v6f7b major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e640effb-1ff4-40f2-8d7b-53732edcd2bf/volumes/kubernetes.io~projected/kube-api-access-r598w:{mountpoint:/var/lib/kubelet/pods/e640effb-1ff4-40f2-8d7b-53732edcd2bf/volumes/kubernetes.io~projected/kube-api-access-r598w major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f1fb4abc-b161-461c-b8ef-bd1212010510/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/f1fb4abc-b161-461c-b8ef-bd1212010510/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f1fb4abc-b161-461c-b8ef-bd1212010510/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/f1fb4abc-b161-461c-b8ef-bd1212010510/volumes/kubernetes.io~empty-dir/tmp major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f1fb4abc-b161-461c-b8ef-bd1212010510/volumes/kubernetes.io~projected/kube-api-access-zb526:{mountpoint:/var/lib/kubelet/pods/f1fb4abc-b161-461c-b8ef-bd1212010510/volumes/kubernetes.io~projected/kube-api-access-zb526 major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4acf40a-d5b0-46e2-b87f-3b96fc9678c4/volumes/kubernetes.io~projected/kube-api-access-xdh8w:{mountpoint:/var/lib/kubelet/pods/f4acf40a-d5b0-46e2-b87f-3b96fc9678c4/volumes/kubernetes.io~projected/kube-api-access-xdh8w major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f93ee017-ab79-447c-90e0-5b85b71dba24/volumes/kubernetes.io~projected/kube-api-access-2lkrg:{mountpoint:/var/lib/kubelet/pods/f93ee017-ab79-447c-90e0-5b85b71dba24/volumes/kubernetes.io~projected/kube-api-access-2lkrg major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f93ee017-ab79-447c-90e0-5b85b71dba24/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/f93ee017-ab79-447c-90e0-5b85b71dba24/volumes/kubernetes.io~secret/serving-cert major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/feda4324-4a5f-491b-a04c-202c222261a0/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/feda4324-4a5f-491b-a04c-202c222261a0/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/feda4324-4a5f-491b-a04c-202c222261a0/volumes/kubernetes.io~projected/kube-api-access-ncqh7:{mountpoint:/var/lib/kubelet/pods/feda4324-4a5f-491b-a04c-202c222261a0/volumes/kubernetes.io~projected/kube-api-access-ncqh7 major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/feda4324-4a5f-491b-a04c-202c222261a0/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/feda4324-4a5f-491b-a04c-202c222261a0/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/feda4324-4a5f-491b-a04c-202c222261a0/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/feda4324-4a5f-491b-a04c-202c222261a0/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:209 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/99d317cb8d20d410394af25a0dc3a863193771ca6f61d7b9464b2105ad94052b/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/9d7a80afd115b8afd9c8e7356af27b27beb677e934a04bc4da252a290ce48843/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/3771b2127562c9f286b8691ccf98c948643ffd091287e1608fcefa8d90efaf82/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/4e1ea6441ef0d33cf51723ef4e6bee2967a57d78769fd50a3ce63fa4295c0bce/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/285ec183dd469c2a878ddcaf344aec0b3baeb7a744134eb83c36eddf3638ee1d/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/4f3427dd66114af44c8c36914af8e3c24c86113215fb7203095a10650c8960f2/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/d1e21e865fe4e58d410b733a7c170f5f3ba52425ccc75b0d987f3a682791672c/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/2acb0aa3611dafaacd16dadf8d4fc38ba552ca2845c62f6e88eae80e34636793/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/6d7888fbcb739a8a0a0199ce9084e2290052194e1ef9216c88a21109e42aacea/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/e018d247e17eb7db407caca537ed0e26f80b3612db22419f7feda4108727ebaf/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/02adac73f4ecb6fe8b20a1faf1d07c4ba83c7a1055d062422d34137085723e2d/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/45f63e3aed01c7d26949db5963147b5f07b8b681a15e9038ca5a95d99555e384/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/3814a09c7eec560fda0a4eab58753b4921b3c6969e16df4b1249a8d52919b3df/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/d805a53e20fc263d44285bb7556d4f54765b509a0bbda62df668f39e25350b13/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/5aa459eef7674bf2dcb0dc3af603a92f9dfb73dab20ce4342882bd1447b3e3bc/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/77ef315e3ce4064eb1d4bc8c3239453e20a667b812171afb0ddc8ecd141cf6a6/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/55f8ca3679cb9ade821d7abdb9d66e5339e092ac12f794f457a22caaa10240c0/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/a06e5ed1ffa5cdb58920be6cd3e00c76853ef1dc1ad7a19c4fb894722d70776c/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/f49bc2ac80da1b68becd3eb77593c19d4b101d0afddd00f99989ac7936e39f17/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/68511659e9bd5d7d56911c105808f7f0267a8773d3cd9e2cc8e7a7a128fc4761/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/ceeef7bf438e084bde64bc41825e228fe5504c2fb6993abe7131b50616a53ed7/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/fe382c9941e79768e348692ea588bcbe802cc06c4349dceeb47fcbbcef06e626/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/01bb1becc8518dac962ec6d3f32d7c2ecd76f6cbadf06e54426be6285f4f0f36/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/078c0e2203764072865b73b56883628a0c49f953c88af7b083a76b9a32e64922/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/621607d9f5272e234e6e75c0b9a64a0b3a276b3b0340991faeef2261cb2cd785/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/7928bbde38d0b2f6690badc8e725feb92c0d183f6ae264cbd2ad696572e0d12b/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/83e82a23b8aa56c9639bba7ddbac732e8dd178c8ab6ddde65c98594df6684492/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/65e192c8b8cd93d582b468c0cf812f935381e81f5c568adbd75c632da50462a3/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/3e65d562f3b548e0525e3b8c0a49104b1a9400d0fb49eee2f3bd97354e666376/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/cd48a13fe7c2f97c424c6adb055a8f26ac5d4588e48fad58fcaf591336a9e8d6/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/da5f88f6cad3175fefcbd40af4253853d537d553bebc81711a9c100e9ddaccf4/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/b697d4d0c46c22cb39b0e65651d03b1658b8aabcdbddfdf1fc40d1ae63cd1dca/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/37122823f4d1e86331ce57166118475d1c4e2dd51ea318fbc87dcddfbfaf1afe/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/8ea0fc7a45c3978ac2c2126a92ace7a27f33d3bcad496e35a1ee1090f5273732/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/f6dc848d8614dd8cacc7019fad5a631fdd0d5b0d46165ddc38b543b7d3070f51/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/903950ca78ef03d7e9e6cd932257c671f11503c86984953355ba5a8b206a4963/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/ee08792552e0b818e3e9436a03553ed8157564321a360929cec8048d4a55c1c0/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/122710f63f51acb3026b485f4a96ddfa460cf2ea3d08b45cf16c9562d22cc40b/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/494c9e34a5cb9efe8142d13a6d2975d2a51701fda8f0bc6570d0f3124b07f34f/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/e3c61d4c0c6d867c1e368ca9080aa8be504a5b665f019cce2e4b234d66b0368a/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/efbff0adac20a46a282e2f39acd14b2fce5a822c1e05252a52a5ec55424f33e7/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/dc32df7f1089f3fd947fca402b6f4a1199ffb3cb9c83f1455f8e3a9b81930a14/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/8059e346174707ce95064ac28db4afcefefdb9e50d6fe52c0b12f9b51866f267/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-339:{mountpoint:/var/lib/containers/storage/overlay/f6fbb51e2966b6f5c22e7875a4869fb2e112f9a15b348fe29f96876509d89c68/merged major:0 minor:339 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/58a81a76a51df6403b0be9e8cdc07a8e5753d45623285b0473daa968711d1e6e/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/feea2322aaae65e0d6bd20c1ae4e760636ddf7d8348922cea2697615081bb7dc/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/281086b079aab98929c6e85a2fef8627e3854e6445f6f62d7bc1461f387edb29/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/0d83219cf329de36983fe51455853a7d45aedef1d8ea70e8df66fd32f79192a5/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/36a5e718211d8597d0635e3d04a2519ff95dbcb6cf0f04398b224ae5247990b0/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/d44efe9bb090cd5eb96aa05764276e3c8f703a1ef6747158bd3554674fbc6fb5/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/e364f3eb45e8a49d45fe9a615bc34646f945ccb19b984b7346a6e78237164211/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/384ca200d682a2aca654a27115aab3405ac95ac6824c97307d78192e71074670/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/712f87b71a35727f664e8a50071cc9b35e6a2fa74394a8897bc553c6e0db71f5/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/4e0ce555854f3b591938823e184d5e43eec6ee6201b694d6ac19e11820155e0e/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/fa99ee6ebbfb1c60131cb678852b688c051868a8a064292154289b4edabbf334/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-75:{mountpoint:/var/lib/containers/storage/overlay/bdca8756957a93fef769e724f527111c7a9606b179f2cb5d4d940ae5c87ca9f8/merged major:0 minor:75 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/a559b05cbffaa2a44fddab154de4b5227eec6555a718fb869e6d321231e3357a/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/643817e2ee52905f08ea4de5e40deba9311f1a0b02c92a787e253bdd3f075b43/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/d0d3ec2de3f759e79241fd98ace84d9a257a3a65b80a8b707cf8b5b972a58440/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/9d15f24b0df20cb7c047e7a46faa8c1b5cd26601a9ef1933ad217d35c906f43e/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/ed478bdb4933ad851632e0537073ba9f9606d80219414397ddf62c766760d5ca/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/82728999356e054211fc7ff358696c2f4ae545bfc4efecb9b0b0c64fcd6efe61/merged major:0 minor:98 fsType:overlay blockSize:0}]
Oct 30 13:03:00.936736 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.936333    5093 manager.go:217] Machine: {Timestamp:2026-10-30 13:03:00.935828625 +0000 UTC m=+0.083095858 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100197 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22119be52f63be34f6b5f5a3d68e91 SystemUUID:ec22119b-e52f-63be-34f6-b5f5a3d68e91 BootID:cff3fcb7-0044-495e-bdaf-bbea1d4e2f7f Filesystems:[{Device:/var/lib/kubelet/pods/7d29458a-da6c-461c-9941-afa323061f30/volumes/kubernetes.io~projected/kube-api-access-789vz DeviceMajor:0 DeviceMinor:355 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7cca8a39-bee0-4705-bcde-b31060e44944/volumes/kubernetes.io~projected/kube-api-access-kncj9 DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/79664154-5454-4294-baad-12bdfad6fe68/volumes/kubernetes.io~projected/kube-api-access-7nrlb DeviceMajor:0 DeviceMinor:233 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a32a253a-ecaf-4f90-af1c-27a7e53b9629/volumes/kubernetes.io~projected/kube-api-access-qsfql DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4e3de8cc64e483689b44b655b9983ba4b90579eb7514936c00f38d8a8d37bed6/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a9872fe8d3163787d5f6fcf7d35449ec1edca5abd3d826b8108c491c819a3783/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8021bc0b-4447-4f04-b9bd-5f80e9cf3507/volumes/kubernetes.io~projected/kube-api-access-262nq DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/155bbdd4-b566-467f-88c6-ec5b4b881a09/volumes/kubernetes.io~projected/kube-api-access-lsj7z DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/46bedd0c-7237-4128-be98-69e4474bb535/volumes/kubernetes.io~projected/kube-api-access-kbpg6 DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5cbda802-f819-4ea6-846e-440cea4afed2/volumes/kubernetes.io~projected/kube-api-access-48dsv DeviceMajor:0 DeviceMinor:217 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/feda4324-4a5f-491b-a04c-202c222261a0/volumes/kubernetes.io~projected/kube-api-access-ncqh7 DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/79664154-5454-4294-baad-12bdfad6fe68/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:227 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f93ee017-ab79-447c-90e0-5b85b71dba24/volumes/kubernetes.io~projected/kube-api-access-2lkrg DeviceMajor:0 DeviceMinor:231 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7bbbc7e0-1790-459b-952a-b6b7b4d55113/volumes/kubernetes.io~projected/kube-api-access-8jdqx DeviceMajor:0 DeviceMinor:341 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f93ee017-ab79-447c-90e0-5b85b71dba24/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:229 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7569409b6e188b60e656069548c94067881169acb401a01f609ce63fa445e43c/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f227bcba4afc53a24a6c3631b7e33b3cc84b82b155f54df82731be6b10bc97fc/userdata/shm DeviceMajor:0 DeviceMinor:347 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/beb2b7e4-b43d-42fc-9e8a-ecdfa6994205/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:49 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9da5be1ca0326ab3d9989a5ac8b3610b3b4a0c14a8fb316621a2045016ce29a8/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d5db3ba8-f1de-4fa5-9839-e7e8200da58f/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/de69ec51-fd3a-4bc4-ac17-ff02bc582905/volumes/kubernetes.io~projected/kube-api-access-v6f7b DeviceMajor:0 DeviceMinor:54 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/de331273dc14e36948f28a293bd31355eecd2f4777b203d2a0420a0f26b75b5f/userdata/shm DeviceMajor:0 DeviceMinor:223 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3c7ab011-8fd4-4d6d-b3f2-72d6af11d3e1/volumes/kubernetes.io~projected/kube-api-access-p4kjx DeviceMajor:0 DeviceMinor:334 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3a675ac7-fbe1-449b-accf-bb0456ed0bd3/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:47 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/087058f5-e286-4ac7-8ed8-a280da332878/volumes/kubernetes.io~projected/kube-api-access-6gdf9 DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5a16e7b2-e45d-480a-a7bc-d914640b9867/volumes/kubernetes.io~projected/kube-api-access-z9m9z DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-75 DeviceMajor:0 DeviceMinor:75 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/feda4324-4a5f-491b-a04c-202c222261a0/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f816757e82dd09f5cfa0a0f30bfaf6c873fadd481a17dfde9e290f9900041ffb/userdata/shm DeviceMajor:0 DeviceMinor:221 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0aef0d1cf6af23a8dab50f6eb1d692bf62f8538e1dc9f793a6d8680e303e4732/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/10b774a1-1476-4f91-b6e5-d52b3f45700e/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:286 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a07f5a09-920e-462f-ab6d-7a2ac53d199e/volumes/kubernetes.io~projected/kube-api-access-gz5ls DeviceMajor:0 DeviceMinor:287 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d5db3ba8-f1de-4fa5-9839-e7e8200da58f/volumes/kubernetes.io~projected/kube-api-access-rflhr DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0c8e72ed-d54d-4e6a-bf66-65a16430351d/volumes/kubernetes.io~projected/kube-api-access-dgqhj DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/46ff484b065035d2962579b1722fe86d2b4e9d2e4b5cea97bf770ae859b773bd/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f4acf40a-d5b0-46e2-b87f-3b96fc9678c4/volumes/kubernetes.io~projected/kube-api-access-xdh8w DeviceMajor:0 DeviceMinor:310 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/f1fb4abc-b161-461c-b8ef-bd1212010510/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:48 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f1fb4abc-b161-461c-b8ef-bd1212010510/volumes/kubernetes.io~projected/kube-api-access-zb526 DeviceMajor:0 DeviceMinor:52 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f1de5a011f5645942aa0dd822f5aae0ca5197e9b5b01aacafd9fa403fe7ed36d/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3a675ac7-fbe1-449b-accf-bb0456ed0bd3/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/4ad66a0c-8dc0-40a4-a1b2-28d26b0de94b/volumes/kubernetes.io~projected/kube-api-access-rnx68 DeviceMajor:0 DeviceMinor:230 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d4c6b7cb6278106280fbe5bc2fb62290e5932aa7d28d851f97901a9af05cdb96/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e640effb-1ff4-40f2-8d7b-53732edcd2bf/volumes/kubernetes.io~projected/kube-api-access-r598w DeviceMajor:0 DeviceMinor:235 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dfe8cfde85a2c351da022ee8103df05f789b03b6ed99ffc5b984323d5cac5816/userdata/shm DeviceMajor:0 DeviceMinor:358 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3a675ac7-fbe1-449b-accf-bb0456ed0bd3/volumes/kubernetes.io~projected/kube-api-access-xws2p DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0d0b2655-d52d-4c95-9589-d45237fa1057/volumes/kubernetes.io~projected/kube-api-access-plb6z DeviceMajor:0 DeviceMinor:234 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/feda4324-4a5f-491b-a04c-202c222261a0/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5cbda802-f819-4ea6-846e-440cea4afed2/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/09db080f67588acba375556f890373b7865b100a2a41cf985e15ec97b6c1899b/userdata/shm DeviceMajor:0 DeviceMinor:342 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7d29458a-da6c-461c-9941-afa323061f30/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:354 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f6950ee20e87263fb0dd7718cc81691d24936acf6bb32219f5659b8b23cfefdd/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0b84f92815296a8a34a2754e0fa1c4510032d61e7fd9551ebf454424a49a192f/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1b27ed0977f245135de6ee6d9d6d381ba53831c5ba22d2699bab4c5db56f508c/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-339 DeviceMajor:0 DeviceMinor:339 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a911208b7a7070410d4ca2f43e084cbbbc08ab928ec7df2bb1a3524f92dc8607/userdata/shm DeviceMajor:0 DeviceMinor:73 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2e963d5ab87bb8f8412d3ed93ae6eef220d880e9b5a0350f3f0c8365b9b5d6b2/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/155bbdd4-b566-467f-88c6-ec5b4b881a09/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7ceb0bb1-def2-44a5-9e31-ceffd848125d/volumes/kubernetes.io~projected/kube-api-access-f5k78 DeviceMajor:0 DeviceMinor:311 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/12715619dec36263a869e0698ef7e31b9481effac5627057399505c660155ba4/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d5db3ba8-f1de-4fa5-9839-e7e8200da58f/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/abf08e3d-eec3-4f73-92a6-4ffb691cbd2d/volumes/kubernetes.io~projected/kube-api-access-tfxrl DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3f90007edff3cc0f6929888fcff5cdf01af5714c69ea75da6d164524fa6c4ba6/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c6bb98ae54b0111cebebb0ab298186b328e44754b62c8032028fd66414186c50/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/562b94e335ef4ee49a4a2c8c6eea2920298d30c5531be03ac529afce7dc418e4/userdata/shm DeviceMajor:0 DeviceMinor:219 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a323e312-dd91-486f-985d-92b9ab8cca04/volumes/kubernetes.io~projected/kube-api-access-whwnz DeviceMajor:0 DeviceMinor:51 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/183f23e7-ae80-445b-960c-fff9b410ba5b/volumes/kubernetes.io~projected/kube-api-access-hjgw5 DeviceMajor:0 DeviceMinor:53 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f1fb4abc-b161-461c-b8ef-bd1212010510/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:50 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2f036d26baa035a5f2f97f9c748b0997a23f90fc3a79944f898e4fc7ebf17f73/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/44f513ab-9115-4761-aa8b-c546212ec17b/volumes/kubernetes.io~projected/kube-api-access-vjjr5 DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/feda4324-4a5f-491b-a04c-202c222261a0/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5/volumes/kubernetes.io~projected/kube-api-access-vdrpm DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1a25d5a7-83d4-443b-98ae-fc7b603cfc5f/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:228 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3cc6fafce08f8f3d202026631327bb8e095f97ff186665b5b1996bd0fe1fbc6c/userdata/shm DeviceMajor:0 DeviceMinor:316 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f2aae6460a32ba64304b33d00d3f239e0ad112f520d74613426b5ca429211604/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0c41bc03f6369e26cb0fac4e724d818f33ed871dba80d4d83b87b5bd08b7c28f/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1a25d5a7-83d4-443b-98ae-fc7b603cfc5f/volumes/kubernetes.io~projected/kube-api-access-qk568 DeviceMajor:0 DeviceMinor:232 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:09db080f67588ac MacAddress:5e:5a:fb:36:88:84 Speed:10000 Mtu:8901} {Name:0aef0d1cf6af23a MacAddress:7e:62:3e:5c:3f:71 Speed:10000 Mtu:8901} {Name:0b84f92815296a8 MacAddress:3e:c3:d0:ab:6c:b8 Speed:10000 Mtu:8901} {Name:12715619dec3626 MacAddress:72:d0:d0:3d:de:18 Speed:10000 Mtu:8901} {Name:1b27ed0977f2451 MacAddress:de:49:80:14:99:c2 Speed:10000 Mtu:8901} {Name:3cc6fafce08f8f3 MacAddress:ee:52:c9:e4:8e:38 Speed:10000 Mtu:8901} {Name:3f90007edff3cc0 MacAddress:22:53:c1:7f:0b:f6 Speed:10000 Mtu:8901} {Name:46ff484b065035d MacAddress:1a:be:f1:e6:a0:f6 Speed:10000 Mtu:8901} {Name:562b94e335ef4ee MacAddress:e2:31:57:3c:0a:0d Speed:10000 Mtu:8901} {Name:7569409b6e188b6 MacAddress:d6:6b:d7:70:8d:bf Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:cb:02:b8:db Speed:0 Mtu:9001} {Name:br-int MacAddress:fa:e7:e6:d8:55:7b Speed:0 Mtu:8901} {Name:d4c6b7cb6278106 MacAddress:4a:19:90:9b:4d:40 Speed:10000 Mtu:8901} {Name:de331273dc14e36 MacAddress:8e:16:8d:4a:8c:df Speed:10000 Mtu:8901} {Name:dfe8cfde85a2c35 MacAddress:4a:20:68:33:db:bc Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:cb:02:b8:db Speed:0 Mtu:9001} {Name:f227bcba4afc53a MacAddress:c6:40:42:25:b2:10 Speed:10000 Mtu:8901} {Name:f6950ee20e87263 MacAddress:06:4c:8c:1b:30:82 Speed:10000 Mtu:8901} {Name:f816757e82dd09f MacAddress:5a:04:67:b1:79:05 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:ea:65:27:d9:7b:2e Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:66:a0:e9:62:7a:be Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 13:03:00.936967 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.936742    5093 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 13:03:00.936967 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.936793    5093 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 13:03:00.937111 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.937085    5093 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 13:03:00.937284 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.937112    5093 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-31.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 13:03:00.937334 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.937286    5093 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 13:03:00.937334 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.937295    5093 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 13:03:00.937334 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.937305    5093 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 13:03:00.937334 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.937320    5093 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 13:03:00.937434 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.937340    5093 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 13:03:00.937511 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.937501    5093 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 13:03:00.937624 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.937615    5093 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 13:03:00.937664 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.937650    5093 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 13:03:00.937664 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.937663    5093 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 13:03:00.937720 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.937672    5093 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 13:03:00.937720 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.937681    5093 kubelet.go:405] "Adding apiserver pod source"
Oct 30 13:03:00.937720 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.937689    5093 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 13:03:00.938412 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.938394    5093 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 13:03:00.938412 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.938407    5093 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 13:03:00.938997 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.938979    5093 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 13:03:00.941489 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.941476    5093 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 13:03:00.941489 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.941489    5093 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 13:03:00.941621 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.941612    5093 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 13:03:00.941815 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.941804    5093 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 13:03:00.941848 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.941833    5093 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 13:03:00.941944 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.941930    5093 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 13:03:00.941976 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.941951    5093 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 13:03:00.941976 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.941957    5093 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 13:03:00.942027 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.941977    5093 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 13:03:00.942027 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.941984    5093 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 13:03:00.942027 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.941990    5093 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 13:03:00.942027 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.941996    5093 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 13:03:00.942027 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.942002    5093 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 13:03:00.942027 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.942009    5093 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 13:03:00.942027 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.942015    5093 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 13:03:00.942027 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.942028    5093 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 13:03:00.942367 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.942038    5093 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 13:03:00.942367 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.942064    5093 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 13:03:00.942367 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.942069    5093 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 13:03:00.945744 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.945729    5093 server.go:1263] "Started kubelet"
Oct 30 13:03:00.945834 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.945779    5093 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 13:03:00.946073 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.945998    5093 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 13:03:00.946423 ip-10-0-134-31 systemd[1]: Started Kubernetes Kubelet.
Oct 30 13:03:00.946950 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.946926    5093 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 13:03:00.947488 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.947469    5093 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 13:03:00.949929 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.949910    5093 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 13:03:00.950392 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.950371    5093 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 13:03:00.950392 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.950392    5093 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 13:03:00.950573 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.950556    5093 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 13:03:00.950637 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:00.950605    5093 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-31.ec2.internal\" not found"
Oct 30 13:03:00.952724 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.952705    5093 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 13:03:00.953407 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.953382    5093 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 13:03:00.954198 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.953558    5093 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 12:57:20 +0000 UTC" deadline="2028-04-15 07:30:56.770717192 +0000 UTC"
Oct 30 13:03:00.954277 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.954205    5093 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12786h27m55.816518854s"
Oct 30 13:03:00.956386 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.956355    5093 factory.go:55] Registering systemd factory
Oct 30 13:03:00.956386 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.956379    5093 factory.go:223] Registration of the systemd container factory successfully
Oct 30 13:03:00.956662 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.956611    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de69ec51-fd3a-4bc4-ac17-ff02bc582905" volumeName="kubernetes.io/configmap/de69ec51-fd3a-4bc4-ac17-ff02bc582905-cni-binary-copy" seLinuxMountContext=""
Oct 30 13:03:00.956696 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.956666    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="feda4324-4a5f-491b-a04c-202c222261a0" volumeName="kubernetes.io/secret/feda4324-4a5f-491b-a04c-202c222261a0-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 13:03:00.956696 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.956681    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="155bbdd4-b566-467f-88c6-ec5b4b881a09" volumeName="kubernetes.io/empty-dir/155bbdd4-b566-467f-88c6-ec5b4b881a09-tmp" seLinuxMountContext=""
Oct 30 13:03:00.956750 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.956694    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d29458a-da6c-461c-9941-afa323061f30" volumeName="kubernetes.io/configmap/7d29458a-da6c-461c-9941-afa323061f30-signing-cabundle" seLinuxMountContext=""
Oct 30 13:03:00.956750 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.956707    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a07f5a09-920e-462f-ab6d-7a2ac53d199e" volumeName="kubernetes.io/projected/a07f5a09-920e-462f-ab6d-7a2ac53d199e-kube-api-access-gz5ls" seLinuxMountContext=""
Oct 30 13:03:00.956750 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.956720    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="beb2b7e4-b43d-42fc-9e8a-ecdfa6994205" volumeName="kubernetes.io/secret/beb2b7e4-b43d-42fc-9e8a-ecdfa6994205-agent-certs" seLinuxMountContext=""
Oct 30 13:03:00.956750 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.956731    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="155bbdd4-b566-467f-88c6-ec5b4b881a09" volumeName="kubernetes.io/configmap/155bbdd4-b566-467f-88c6-ec5b4b881a09-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 13:03:00.956750 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.956743    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79664154-5454-4294-baad-12bdfad6fe68" volumeName="kubernetes.io/configmap/79664154-5454-4294-baad-12bdfad6fe68-config" seLinuxMountContext=""
Oct 30 13:03:00.956931 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.956756    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4fd1eb6-625a-4649-ba27-9e8798fd1bc5" volumeName="kubernetes.io/configmap/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-registry-certificates" seLinuxMountContext=""
Oct 30 13:03:00.956931 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.956767    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4fd1eb6-625a-4649-ba27-9e8798fd1bc5" volumeName="kubernetes.io/empty-dir/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-ca-trust-extracted" seLinuxMountContext=""
Oct 30 13:03:00.956931 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.956779    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4acf40a-d5b0-46e2-b87f-3b96fc9678c4" volumeName="kubernetes.io/configmap/f4acf40a-d5b0-46e2-b87f-3b96fc9678c4-service-ca-bundle" seLinuxMountContext=""
Oct 30 13:03:00.956931 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.956790    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4acf40a-d5b0-46e2-b87f-3b96fc9678c4" volumeName="kubernetes.io/projected/f4acf40a-d5b0-46e2-b87f-3b96fc9678c4-kube-api-access-xdh8w" seLinuxMountContext=""
Oct 30 13:03:00.956931 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.956802    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a675ac7-fbe1-449b-accf-bb0456ed0bd3" volumeName="kubernetes.io/configmap/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 13:03:00.956931 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.956814    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10b774a1-1476-4f91-b6e5-d52b3f45700e" volumeName="kubernetes.io/secret/10b774a1-1476-4f91-b6e5-d52b3f45700e-original-pull-secret" seLinuxMountContext=""
Oct 30 13:03:00.956931 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.956830    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7bbbc7e0-1790-459b-952a-b6b7b4d55113" volumeName="kubernetes.io/projected/7bbbc7e0-1790-459b-952a-b6b7b4d55113-kube-api-access-8jdqx" seLinuxMountContext=""
Oct 30 13:03:00.956931 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.956844    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4acf40a-d5b0-46e2-b87f-3b96fc9678c4" volumeName="kubernetes.io/configmap/f4acf40a-d5b0-46e2-b87f-3b96fc9678c4-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 13:03:00.956931 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.956857    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f93ee017-ab79-447c-90e0-5b85b71dba24" volumeName="kubernetes.io/projected/f93ee017-ab79-447c-90e0-5b85b71dba24-kube-api-access-2lkrg" seLinuxMountContext=""
Oct 30 13:03:00.956931 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.956869    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="feda4324-4a5f-491b-a04c-202c222261a0" volumeName="kubernetes.io/empty-dir/feda4324-4a5f-491b-a04c-202c222261a0-ca-trust-extracted" seLinuxMountContext=""
Oct 30 13:03:00.956931 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.956882    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="183f23e7-ae80-445b-960c-fff9b410ba5b" volumeName="kubernetes.io/configmap/183f23e7-ae80-445b-960c-fff9b410ba5b-cni-binary-copy" seLinuxMountContext=""
Oct 30 13:03:00.956931 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.956894    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a25d5a7-83d4-443b-98ae-fc7b603cfc5f" volumeName="kubernetes.io/secret/1a25d5a7-83d4-443b-98ae-fc7b603cfc5f-cert" seLinuxMountContext=""
Oct 30 13:03:00.956931 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.956905    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d29458a-da6c-461c-9941-afa323061f30" volumeName="kubernetes.io/projected/7d29458a-da6c-461c-9941-afa323061f30-kube-api-access-789vz" seLinuxMountContext=""
Oct 30 13:03:00.956931 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.956916    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de69ec51-fd3a-4bc4-ac17-ff02bc582905" volumeName="kubernetes.io/projected/de69ec51-fd3a-4bc4-ac17-ff02bc582905-kube-api-access-v6f7b" seLinuxMountContext=""
Oct 30 13:03:00.956931 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.956928    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1fb4abc-b161-461c-b8ef-bd1212010510" volumeName="kubernetes.io/projected/f1fb4abc-b161-461c-b8ef-bd1212010510-kube-api-access-zb526" seLinuxMountContext=""
Oct 30 13:03:00.956931 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.956940    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d0b2655-d52d-4c95-9589-d45237fa1057" volumeName="kubernetes.io/configmap/0d0b2655-d52d-4c95-9589-d45237fa1057-config-volume" seLinuxMountContext=""
Oct 30 13:03:00.957694 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.956951    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a25d5a7-83d4-443b-98ae-fc7b603cfc5f" volumeName="kubernetes.io/projected/1a25d5a7-83d4-443b-98ae-fc7b603cfc5f-kube-api-access-qk568" seLinuxMountContext=""
Oct 30 13:03:00.957694 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.956966    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a675ac7-fbe1-449b-accf-bb0456ed0bd3" volumeName="kubernetes.io/configmap/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-ovnkube-config" seLinuxMountContext=""
Oct 30 13:03:00.957694 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.956988    5093 factory.go:153] Registering CRI-O factory
Oct 30 13:03:00.957694 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.957010    5093 factory.go:223] Registration of the crio container factory successfully
Oct 30 13:03:00.957694 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.957068    5093 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 13:03:00.957694 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.957088    5093 factory.go:103] Registering Raw factory
Oct 30 13:03:00.957694 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.957104    5093 manager.go:1199] Started watching for new ooms in manager
Oct 30 13:03:00.957694 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.956977    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de69ec51-fd3a-4bc4-ac17-ff02bc582905" volumeName="kubernetes.io/configmap/de69ec51-fd3a-4bc4-ac17-ff02bc582905-multus-daemon-config" seLinuxMountContext=""
Oct 30 13:03:00.957694 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.957466    5093 manager.go:319] Starting recovery of all containers
Oct 30 13:03:00.957694 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.957474    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c8e72ed-d54d-4e6a-bf66-65a16430351d" volumeName="kubernetes.io/configmap/0c8e72ed-d54d-4e6a-bf66-65a16430351d-iptables-alerter-script" seLinuxMountContext=""
Oct 30 13:03:00.957694 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.957506    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a675ac7-fbe1-449b-accf-bb0456ed0bd3" volumeName="kubernetes.io/configmap/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-env-overrides" seLinuxMountContext=""
Oct 30 13:03:00.957694 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.957520    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5cbda802-f819-4ea6-846e-440cea4afed2" volumeName="kubernetes.io/projected/5cbda802-f819-4ea6-846e-440cea4afed2-kube-api-access-48dsv" seLinuxMountContext=""
Oct 30 13:03:00.957694 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.957532    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cca8a39-bee0-4705-bcde-b31060e44944" volumeName="kubernetes.io/configmap/7cca8a39-bee0-4705-bcde-b31060e44944-samples-operator-config" seLinuxMountContext=""
Oct 30 13:03:00.957694 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.957544    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cca8a39-bee0-4705-bcde-b31060e44944" volumeName="kubernetes.io/projected/7cca8a39-bee0-4705-bcde-b31060e44944-kube-api-access-kncj9" seLinuxMountContext=""
Oct 30 13:03:00.957694 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.957555    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ceb0bb1-def2-44a5-9e31-ceffd848125d" volumeName="kubernetes.io/empty-dir/7ceb0bb1-def2-44a5-9e31-ceffd848125d-data-volume" seLinuxMountContext=""
Oct 30 13:03:00.957694 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.957568    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a323e312-dd91-486f-985d-92b9ab8cca04" volumeName="kubernetes.io/projected/a323e312-dd91-486f-985d-92b9ab8cca04-kube-api-access-whwnz" seLinuxMountContext=""
Oct 30 13:03:00.957694 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.957579    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4fd1eb6-625a-4649-ba27-9e8798fd1bc5" volumeName="kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-kube-api-access-vdrpm" seLinuxMountContext=""
Oct 30 13:03:00.957694 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.957591    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e640effb-1ff4-40f2-8d7b-53732edcd2bf" volumeName="kubernetes.io/configmap/e640effb-1ff4-40f2-8d7b-53732edcd2bf-telemetry-config" seLinuxMountContext=""
Oct 30 13:03:00.957694 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.957602    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="155bbdd4-b566-467f-88c6-ec5b4b881a09" volumeName="kubernetes.io/empty-dir/155bbdd4-b566-467f-88c6-ec5b4b881a09-snapshots" seLinuxMountContext=""
Oct 30 13:03:00.957694 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.957613    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46bedd0c-7237-4128-be98-69e4474bb535" volumeName="kubernetes.io/projected/46bedd0c-7237-4128-be98-69e4474bb535-kube-api-access-kbpg6" seLinuxMountContext=""
Oct 30 13:03:00.957694 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.957625    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a16e7b2-e45d-480a-a7bc-d914640b9867" volumeName="kubernetes.io/empty-dir/5a16e7b2-e45d-480a-a7bc-d914640b9867-tmp-dir" seLinuxMountContext=""
Oct 30 13:03:00.957694 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.957645    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abf08e3d-eec3-4f73-92a6-4ffb691cbd2d" volumeName="kubernetes.io/projected/abf08e3d-eec3-4f73-92a6-4ffb691cbd2d-kube-api-access-tfxrl" seLinuxMountContext=""
Oct 30 13:03:00.957694 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.957668    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5db3ba8-f1de-4fa5-9839-e7e8200da58f" volumeName="kubernetes.io/projected/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-kube-api-access-rflhr" seLinuxMountContext=""
Oct 30 13:03:00.957694 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.957681    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4acf40a-d5b0-46e2-b87f-3b96fc9678c4" volumeName="kubernetes.io/empty-dir/f4acf40a-d5b0-46e2-b87f-3b96fc9678c4-archives-path" seLinuxMountContext=""
Oct 30 13:03:00.957694 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.957693    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="feda4324-4a5f-491b-a04c-202c222261a0" volumeName="kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-bound-sa-token" seLinuxMountContext=""
Oct 30 13:03:00.957694 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.957706    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a675ac7-fbe1-449b-accf-bb0456ed0bd3" volumeName="kubernetes.io/projected/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-kube-api-access-xws2p" seLinuxMountContext=""
Oct 30 13:03:00.958959 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.957718    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63086243-e82d-4fc5-bfef-e4241c22ad4e" volumeName="kubernetes.io/configmap/63086243-e82d-4fc5-bfef-e4241c22ad4e-nginx-conf" seLinuxMountContext=""
Oct 30 13:03:00.958959 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.957731    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4fd1eb6-625a-4649-ba27-9e8798fd1bc5" volumeName="kubernetes.io/secret/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 13:03:00.958959 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.957743    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d0b2655-d52d-4c95-9589-d45237fa1057" volumeName="kubernetes.io/projected/0d0b2655-d52d-4c95-9589-d45237fa1057-kube-api-access-plb6z" seLinuxMountContext=""
Oct 30 13:03:00.958959 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.957754    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a16e7b2-e45d-480a-a7bc-d914640b9867" volumeName="kubernetes.io/projected/5a16e7b2-e45d-480a-a7bc-d914640b9867-kube-api-access-z9m9z" seLinuxMountContext=""
Oct 30 13:03:00.958959 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.957766    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8021bc0b-4447-4f04-b9bd-5f80e9cf3507" volumeName="kubernetes.io/projected/8021bc0b-4447-4f04-b9bd-5f80e9cf3507-kube-api-access-262nq" seLinuxMountContext=""
Oct 30 13:03:00.958959 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.957777    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a32a253a-ecaf-4f90-af1c-27a7e53b9629" volumeName="kubernetes.io/projected/a32a253a-ecaf-4f90-af1c-27a7e53b9629-kube-api-access-qsfql" seLinuxMountContext=""
Oct 30 13:03:00.958959 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.957789    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1fb4abc-b161-461c-b8ef-bd1212010510" volumeName="kubernetes.io/empty-dir/f1fb4abc-b161-461c-b8ef-bd1212010510-tmp" seLinuxMountContext=""
Oct 30 13:03:00.958959 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.957802    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f93ee017-ab79-447c-90e0-5b85b71dba24" volumeName="kubernetes.io/configmap/f93ee017-ab79-447c-90e0-5b85b71dba24-trusted-ca" seLinuxMountContext=""
Oct 30 13:03:00.958959 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.957814    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="feda4324-4a5f-491b-a04c-202c222261a0" volumeName="kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-kube-api-access-ncqh7" seLinuxMountContext=""
Oct 30 13:03:00.958959 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.957826    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c7ab011-8fd4-4d6d-b3f2-72d6af11d3e1" volumeName="kubernetes.io/projected/3c7ab011-8fd4-4d6d-b3f2-72d6af11d3e1-kube-api-access-p4kjx" seLinuxMountContext=""
Oct 30 13:03:00.958959 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.957840    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="087058f5-e286-4ac7-8ed8-a280da332878" volumeName="kubernetes.io/projected/087058f5-e286-4ac7-8ed8-a280da332878-kube-api-access-6gdf9" seLinuxMountContext=""
Oct 30 13:03:00.958959 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.957852    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d0b2655-d52d-4c95-9589-d45237fa1057" volumeName="kubernetes.io/empty-dir/0d0b2655-d52d-4c95-9589-d45237fa1057-tmp-dir" seLinuxMountContext=""
Oct 30 13:03:00.958959 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.957877    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="155bbdd4-b566-467f-88c6-ec5b4b881a09" volumeName="kubernetes.io/configmap/155bbdd4-b566-467f-88c6-ec5b4b881a09-service-ca-bundle" seLinuxMountContext=""
Oct 30 13:03:00.958959 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.957891    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="183f23e7-ae80-445b-960c-fff9b410ba5b" volumeName="kubernetes.io/configmap/183f23e7-ae80-445b-960c-fff9b410ba5b-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 13:03:00.958959 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.957903    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ceb0bb1-def2-44a5-9e31-ceffd848125d" volumeName="kubernetes.io/configmap/7ceb0bb1-def2-44a5-9e31-ceffd848125d-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 30 13:03:00.958959 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.957920    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="beb2b7e4-b43d-42fc-9e8a-ecdfa6994205" volumeName="kubernetes.io/configmap/beb2b7e4-b43d-42fc-9e8a-ecdfa6994205-konnectivity-ca" seLinuxMountContext=""
Oct 30 13:03:00.958959 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.957933    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="feda4324-4a5f-491b-a04c-202c222261a0" volumeName="kubernetes.io/configmap/feda4324-4a5f-491b-a04c-202c222261a0-trusted-ca" seLinuxMountContext=""
Oct 30 13:03:00.958959 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.957954    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="183f23e7-ae80-445b-960c-fff9b410ba5b" volumeName="kubernetes.io/projected/183f23e7-ae80-445b-960c-fff9b410ba5b-kube-api-access-hjgw5" seLinuxMountContext=""
Oct 30 13:03:00.958959 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.957965    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5cbda802-f819-4ea6-846e-440cea4afed2" volumeName="kubernetes.io/configmap/5cbda802-f819-4ea6-846e-440cea4afed2-config" seLinuxMountContext=""
Oct 30 13:03:00.958959 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.957979    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79664154-5454-4294-baad-12bdfad6fe68" volumeName="kubernetes.io/projected/79664154-5454-4294-baad-12bdfad6fe68-kube-api-access-7nrlb" seLinuxMountContext=""
Oct 30 13:03:00.958959 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.957992    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79664154-5454-4294-baad-12bdfad6fe68" volumeName="kubernetes.io/secret/79664154-5454-4294-baad-12bdfad6fe68-serving-cert" seLinuxMountContext=""
Oct 30 13:03:00.958959 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.958005    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ceb0bb1-def2-44a5-9e31-ceffd848125d" volumeName="kubernetes.io/projected/7ceb0bb1-def2-44a5-9e31-ceffd848125d-kube-api-access-f5k78" seLinuxMountContext=""
Oct 30 13:03:00.958959 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.958018    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="155bbdd4-b566-467f-88c6-ec5b4b881a09" volumeName="kubernetes.io/projected/155bbdd4-b566-467f-88c6-ec5b4b881a09-kube-api-access-lsj7z" seLinuxMountContext=""
Oct 30 13:03:00.958959 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.958030    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="087058f5-e286-4ac7-8ed8-a280da332878" volumeName="kubernetes.io/configmap/087058f5-e286-4ac7-8ed8-a280da332878-serviceca" seLinuxMountContext=""
Oct 30 13:03:00.958959 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.958043    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5cbda802-f819-4ea6-846e-440cea4afed2" volumeName="kubernetes.io/secret/5cbda802-f819-4ea6-846e-440cea4afed2-serving-cert" seLinuxMountContext=""
Oct 30 13:03:00.958959 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.958056    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5db3ba8-f1de-4fa5-9839-e7e8200da58f" volumeName="kubernetes.io/secret/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-stats-auth" seLinuxMountContext=""
Oct 30 13:03:00.958959 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.958068    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1fb4abc-b161-461c-b8ef-bd1212010510" volumeName="kubernetes.io/empty-dir/f1fb4abc-b161-461c-b8ef-bd1212010510-etc-tuned" seLinuxMountContext=""
Oct 30 13:03:00.958959 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.958080    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f93ee017-ab79-447c-90e0-5b85b71dba24" volumeName="kubernetes.io/configmap/f93ee017-ab79-447c-90e0-5b85b71dba24-config" seLinuxMountContext=""
Oct 30 13:03:00.958959 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.958093    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f93ee017-ab79-447c-90e0-5b85b71dba24" volumeName="kubernetes.io/secret/f93ee017-ab79-447c-90e0-5b85b71dba24-serving-cert" seLinuxMountContext=""
Oct 30 13:03:00.958959 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.958107    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="feda4324-4a5f-491b-a04c-202c222261a0" volumeName="kubernetes.io/configmap/feda4324-4a5f-491b-a04c-202c222261a0-registry-certificates" seLinuxMountContext=""
Oct 30 13:03:00.958959 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.958120    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a675ac7-fbe1-449b-accf-bb0456ed0bd3" volumeName="kubernetes.io/secret/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 13:03:00.958959 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.958158    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4fd1eb6-625a-4649-ba27-9e8798fd1bc5" volumeName="kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-bound-sa-token" seLinuxMountContext=""
Oct 30 13:03:00.958959 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.958171    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e640effb-1ff4-40f2-8d7b-53732edcd2bf" volumeName="kubernetes.io/projected/e640effb-1ff4-40f2-8d7b-53732edcd2bf-kube-api-access-r598w" seLinuxMountContext=""
Oct 30 13:03:00.958959 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.958184    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="feda4324-4a5f-491b-a04c-202c222261a0" volumeName="kubernetes.io/secret/feda4324-4a5f-491b-a04c-202c222261a0-installation-pull-secrets" seLinuxMountContext=""
Oct 30 13:03:00.958959 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.958195    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="155bbdd4-b566-467f-88c6-ec5b4b881a09" volumeName="kubernetes.io/secret/155bbdd4-b566-467f-88c6-ec5b4b881a09-serving-cert" seLinuxMountContext=""
Oct 30 13:03:00.958959 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.958238    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c8e72ed-d54d-4e6a-bf66-65a16430351d" volumeName="kubernetes.io/projected/0c8e72ed-d54d-4e6a-bf66-65a16430351d-kube-api-access-dgqhj" seLinuxMountContext=""
Oct 30 13:03:00.958959 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.958255    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="183f23e7-ae80-445b-960c-fff9b410ba5b" volumeName="kubernetes.io/configmap/183f23e7-ae80-445b-960c-fff9b410ba5b-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 13:03:00.958959 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.958272    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ad66a0c-8dc0-40a4-a1b2-28d26b0de94b" volumeName="kubernetes.io/projected/4ad66a0c-8dc0-40a4-a1b2-28d26b0de94b-kube-api-access-rnx68" seLinuxMountContext=""
Oct 30 13:03:00.958959 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.958286    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d29458a-da6c-461c-9941-afa323061f30" volumeName="kubernetes.io/secret/7d29458a-da6c-461c-9941-afa323061f30-signing-key" seLinuxMountContext=""
Oct 30 13:03:00.958959 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.958299    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5db3ba8-f1de-4fa5-9839-e7e8200da58f" volumeName="kubernetes.io/secret/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-default-certificate" seLinuxMountContext=""
Oct 30 13:03:00.958959 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.958311    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44f513ab-9115-4761-aa8b-c546212ec17b" volumeName="kubernetes.io/projected/44f513ab-9115-4761-aa8b-c546212ec17b-kube-api-access-vjjr5" seLinuxMountContext=""
Oct 30 13:03:00.958959 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.958327    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4fd1eb6-625a-4649-ba27-9e8798fd1bc5" volumeName="kubernetes.io/configmap/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-trusted-ca" seLinuxMountContext=""
Oct 30 13:03:00.958959 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.958340    5093 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4fd1eb6-625a-4649-ba27-9e8798fd1bc5" volumeName="kubernetes.io/secret/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-installation-pull-secrets" seLinuxMountContext=""
Oct 30 13:03:00.958959 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.958351    5093 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 13:03:00.958959 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.958358    5093 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 13:03:00.980103 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.980078    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jdj4_f93ee017-ab79-447c-90e0-5b85b71dba24/console-operator/1.log"
Oct 30 13:03:00.980820 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.980676    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jdj4_f93ee017-ab79-447c-90e0-5b85b71dba24/console-operator/1.log"
Oct 30 13:03:00.993763 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.988896    5093 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 13:03:00.993763 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.990543    5093 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 13:03:00.993763 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.990567    5093 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 13:03:00.993763 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.990586    5093 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 13:03:00.993763 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:00.990632    5093 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 13:03:00.995556 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.995481    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tnm_3a675ac7-fbe1-449b-accf-bb0456ed0bd3/ovn-acl-logging/0.log"
Oct 30 13:03:00.997302 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:00.997282    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tnm_3a675ac7-fbe1-449b-accf-bb0456ed0bd3/ovn-acl-logging/0.log"
Oct 30 13:03:01.005564 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.005540    5093 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 13:03:01.006436 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.006411    5093 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 13:03:01.014024 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.014003    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tnm_3a675ac7-fbe1-449b-accf-bb0456ed0bd3/ovn-acl-logging/0.log"
Oct 30 13:03:01.014884 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.014853    5093 generic.go:356] "Generic (PLEG): container finished" podID="3a675ac7-fbe1-449b-accf-bb0456ed0bd3" containerID="b515d20ead38e58fcfaaa3b97b85952de70b11decbbdc1ca22cf2b35fe70670c" exitCode=1
Oct 30 13:03:01.017089 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.017068    5093 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 13:03:01.018758 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.018737    5093 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 13:03:01.018980 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.018961    5093 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 13:03:01.048656 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.048625    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jdj4_f93ee017-ab79-447c-90e0-5b85b71dba24/console-operator/1.log"
Oct 30 13:03:01.048816 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.048665    5093 generic.go:356] "Generic (PLEG): container finished" podID="f93ee017-ab79-447c-90e0-5b85b71dba24" containerID="843f19531f8ed50dca559ae32fd0ed454c48aa876303554a6b13044481fee1a7" exitCode=255
Oct 30 13:03:01.055628 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.055592    5093 generic.go:356] "Generic (PLEG): container finished" podID="183f23e7-ae80-445b-960c-fff9b410ba5b" containerID="a04266638b5538f4488b3a0a1d5050b776751dbab333940f5d123974e80382b6" exitCode=0
Oct 30 13:03:01.055628 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.055627    5093 generic.go:356] "Generic (PLEG): container finished" podID="183f23e7-ae80-445b-960c-fff9b410ba5b" containerID="b94baaa2927291a4b20a262129ceaf5309bbfe071d8a90f62f2e6b53d1c81c81" exitCode=0
Oct 30 13:03:01.055844 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.055640    5093 generic.go:356] "Generic (PLEG): container finished" podID="183f23e7-ae80-445b-960c-fff9b410ba5b" containerID="c758bf56aa652a62088df4a941716f0043730bd5e00586c7eba31a97f25e8092" exitCode=0
Oct 30 13:03:01.055844 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.055650    5093 generic.go:356] "Generic (PLEG): container finished" podID="183f23e7-ae80-445b-960c-fff9b410ba5b" containerID="2a8648ce69c555e4848b8163c06040a075d2defd78298a829108a0e7d2efb315" exitCode=0
Oct 30 13:03:01.055844 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.055660    5093 generic.go:356] "Generic (PLEG): container finished" podID="183f23e7-ae80-445b-960c-fff9b410ba5b" containerID="62fbb027596ba7b31ae1480cfd43f7c21605927721260e28537ce8164858fc50" exitCode=0
Oct 30 13:03:01.055844 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.055670    5093 generic.go:356] "Generic (PLEG): container finished" podID="183f23e7-ae80-445b-960c-fff9b410ba5b" containerID="3dbef6f614e977c11d58740c2274d9ac1c89bd6ab385fd78cc465502e249bc46" exitCode=0
Oct 30 13:03:01.059555 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.059527    5093 generic.go:356] "Generic (PLEG): container finished" podID="87f9e67ec9b4ec6ee729521e7847a5a2" containerID="373f8b51d46bf6fe5f41a142dc388ae8a641cd6b7d0bd42bfcc161e8d6608e65" exitCode=0
Oct 30 13:03:01.066095 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.066075    5093 manager.go:324] Recovery completed
Oct 30 13:03:01.087985 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.087954    5093 cpu_manager.go:226] "Starting" policy="none"
Oct 30 13:03:01.087985 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.087980    5093 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 13:03:01.088234 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.088006    5093 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 13:03:01.088285 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.088229    5093 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 13:03:01.088285 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.088265    5093 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 13:03:01.088366 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.088290    5093 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 13:03:01.088366 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.088298    5093 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 13:03:01.088366 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.088306    5093 policy_none.go:50] "Start"
Oct 30 13:03:01.088366 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.088341    5093 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 13:03:01.088366 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.088354    5093 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 13:03:01.088588 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.088573    5093 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 13:03:01.088640 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.088588    5093 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 13:03:01.088640 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.088597    5093 policy_none.go:44] "Start"
Oct 30 13:03:01.090700 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:01.090678    5093 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 30 13:03:01.093636 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.093606    5093 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 13:03:01.096918 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.096906    5093 manager.go:341] "Starting Device Plugin manager"
Oct 30 13:03:01.097483 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:01.096941    5093 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 13:03:01.097524 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.097488    5093 server.go:87] "Starting device plugin registration server"
Oct 30 13:03:01.097762 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.097751    5093 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 13:03:01.097810 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.097765    5093 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 13:03:01.097887 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.097866    5093 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 13:03:01.098199 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.097986    5093 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 13:03:01.098199 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.098031    5093 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 13:03:01.098199 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.098038    5093 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 13:03:01.098199 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.098076    5093 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T13:03:01.098019981Z","UUID":"02275e09-9f83-4fad-896c-cd8026e8d8b5","Handler":null,"Name":"","Endpoint":""}
Oct 30 13:03:01.099362 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:01.099342    5093 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 13:03:01.100305 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.100287    5093 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 13:03:01.100411 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.100314    5093 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 13:03:01.198458 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.198426    5093 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 13:03:01.199705 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.199686    5093 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-31.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 13:03:01.199812 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.199716    5093 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-31.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 13:03:01.199812 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.199726    5093 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-31.ec2.internal" event="NodeHasSufficientPID"
Oct 30 13:03:01.199812 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.199775    5093 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-31.ec2.internal"
Oct 30 13:03:01.234989 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.234962    5093 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-134-31.ec2.internal"
Oct 30 13:03:01.235117 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.235048    5093 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-31.ec2.internal"
Oct 30 13:03:01.291237 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.291184    5093 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-31.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-31.ec2.internal"]
Oct 30 13:03:01.291874 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.291830    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-31.ec2.internal" event={"ID":"5213eb0d1df39bb19f1777f20f292baf","Type":"ContainerStarted","Data":"0018ddbe4e6d2cdcf0b570d1855a68a1f076c0856c42d4940fa6b1e725e6c986"}
Oct 30 13:03:01.291960 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.291885    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-31.ec2.internal" event={"ID":"5213eb0d1df39bb19f1777f20f292baf","Type":"ContainerStarted","Data":"9da5be1ca0326ab3d9989a5ac8b3610b3b4a0c14a8fb316621a2045016ce29a8"}
Oct 30 13:03:01.291960 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.291952    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-31.ec2.internal" event={"ID":"87f9e67ec9b4ec6ee729521e7847a5a2","Type":"ContainerStarted","Data":"6e39d68b4ab004a78973391e1a1585e868d71d91f7dfd95d6c2aa4562d035695"}
Oct 30 13:03:01.292027 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.291960    5093 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-31.ec2.internal"
Oct 30 13:03:01.292027 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.291972    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-31.ec2.internal" event={"ID":"87f9e67ec9b4ec6ee729521e7847a5a2","Type":"ContainerDied","Data":"373f8b51d46bf6fe5f41a142dc388ae8a641cd6b7d0bd42bfcc161e8d6608e65"}
Oct 30 13:03:01.292027 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.291991    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-31.ec2.internal" event={"ID":"87f9e67ec9b4ec6ee729521e7847a5a2","Type":"ContainerStarted","Data":"f2aae6460a32ba64304b33d00d3f239e0ad112f520d74613426b5ca429211604"}
Oct 30 13:03:01.292241 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.292078    5093 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-31.ec2.internal"
Oct 30 13:03:01.317113 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.317079    5093 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 13:03:01.317297 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:01.317157    5093 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-31.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-31.ec2.internal"
Oct 30 13:03:01.317519 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.317507    5093 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 13:03:01.317573 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:01.317539    5093 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-31.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-31.ec2.internal"
Oct 30 13:03:01.407918 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.407889    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/87f9e67ec9b4ec6ee729521e7847a5a2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-31.ec2.internal\" (UID: \"87f9e67ec9b4ec6ee729521e7847a5a2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-31.ec2.internal"
Oct 30 13:03:01.407918 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.407922    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87f9e67ec9b4ec6ee729521e7847a5a2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-31.ec2.internal\" (UID: \"87f9e67ec9b4ec6ee729521e7847a5a2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-31.ec2.internal"
Oct 30 13:03:01.408116 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.407943    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5213eb0d1df39bb19f1777f20f292baf-config\") pod \"kube-apiserver-proxy-ip-10-0-134-31.ec2.internal\" (UID: \"5213eb0d1df39bb19f1777f20f292baf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-31.ec2.internal"
Oct 30 13:03:01.508597 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.508499    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87f9e67ec9b4ec6ee729521e7847a5a2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-31.ec2.internal\" (UID: \"87f9e67ec9b4ec6ee729521e7847a5a2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-31.ec2.internal"
Oct 30 13:03:01.508597 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.508585    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87f9e67ec9b4ec6ee729521e7847a5a2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-31.ec2.internal\" (UID: \"87f9e67ec9b4ec6ee729521e7847a5a2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-31.ec2.internal"
Oct 30 13:03:01.508773 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.508607    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5213eb0d1df39bb19f1777f20f292baf-config\") pod \"kube-apiserver-proxy-ip-10-0-134-31.ec2.internal\" (UID: \"5213eb0d1df39bb19f1777f20f292baf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-31.ec2.internal"
Oct 30 13:03:01.508773 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.508639    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/87f9e67ec9b4ec6ee729521e7847a5a2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-31.ec2.internal\" (UID: \"87f9e67ec9b4ec6ee729521e7847a5a2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-31.ec2.internal"
Oct 30 13:03:01.508773 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.508640    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5213eb0d1df39bb19f1777f20f292baf-config\") pod \"kube-apiserver-proxy-ip-10-0-134-31.ec2.internal\" (UID: \"5213eb0d1df39bb19f1777f20f292baf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-31.ec2.internal"
Oct 30 13:03:01.508773 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.508704    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/87f9e67ec9b4ec6ee729521e7847a5a2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-31.ec2.internal\" (UID: \"87f9e67ec9b4ec6ee729521e7847a5a2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-31.ec2.internal"
Oct 30 13:03:01.938366 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.938334    5093 apiserver.go:52] "Watching apiserver"
Oct 30 13:03:01.996172 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.996117    5093 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 13:03:01.996896 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.996868    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q945","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gxcrm","kube-system/kas-connection-checker-685895c7c9-rbf7t","openshift-insights/periodic-gathering-9wlmq-z26dn","kube-system/global-pull-secret-syncer-wrfqz","openshift-ovn-kubernetes/ovnkube-node-67tnm","openshift-image-registry/node-ca-f7mvz","openshift-multus/network-metrics-daemon-mbncl","openshift-service-ca-operator/service-ca-operator-544d749c8c-fhnzt","openshift-cluster-node-tuning-operator/tuned-s9vk9","openshift-kube-storage-version-migrator/migrator-66df5d496b-r5pcv","openshift-network-diagnostics/network-check-source-6f78d59f99-r7vrw","openshift-image-registry/image-registry-5b97bd7c47-dssmx","kube-system/konnectivity-agent-wpqvv","openshift-ingress/router-default-668b54766c-wgk28","kube-system/kas-connection-checker-685895c7c9-22fm4","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lt2qg","kube-system/kube-apiserver-proxy-ip-10-0-134-31.ec2.internal","openshift-kube-storage-version-migrator/migrator-66df5d496b-k6rlj","openshift-monitoring/cluster-monitoring-operator-884576c57-x8wqt","openshift-image-registry/image-registry-9677cf66b-kpdfl","openshift-console-operator/console-operator-7cd56b8dbc-6jdj4","openshift-insights/insights-runtime-extractor-nk2zz","openshift-multus/multus-additional-cni-plugins-lrqnm","openshift-ingress-canary/ingress-canary-wmb4b","openshift-network-operator/iptables-alerter-5swg6","openshift-dns/node-resolver-gltng","openshift-dns/dns-default-8nb7s","openshift-insights/insights-operator-7c4986d449-lgssx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-31.ec2.internal","openshift-service-ca/service-ca-5fffc97bdf-x8ck2","openshift-network-diagnostics/network-check-target-rxd99","openshift-multus/multus-f2bqp","openshift-network-console/networking-console-plugin-86c478bb88-2w8jt","kube-system/kas-connection-checker-685895c7c9-wnzh2","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-lvf7h"]
Oct 30 13:03:01.999221 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:01.999194    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x8wqt"
Oct 30 13:03:02.000453 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.000434    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gxcrm"
Oct 30 13:03:02.000857 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.000834    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-668b54766c-wgk28"
Oct 30 13:03:02.001355 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.001335    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:03:02.002051 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.002035    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-2w8jt"
Oct 30 13:03:02.003556 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.003540    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbncl"
Oct 30 13:03:02.010927 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.010902    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:03:02.011437 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.011422    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8nb7s"
Oct 30 13:03:02.012778 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.012761    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nk2zz"
Oct 30 13:03:02.013623 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.013602    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 13:03:02.013623 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.013608    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 13:03:02.014992 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.014971    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 13:03:02.014992 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.014992    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 13:03:02.015264 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.015007    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 13:03:02.015264 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.015038    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 13:03:02.015443 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.015389    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 13:03:02.015503 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.015462    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 13:03:02.015580 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.015554    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 13:03:02.015699 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.015644    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 13:03:02.015876 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.015853    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 13:03:02.015961 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.015904    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 13:03:02.029519 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.029475    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 13:03:02.029519 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.029476    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 13:03:02.029519 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.029519    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-6cbwz\""
Oct 30 13:03:02.029723 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.029546    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 13:03:02.029723 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.029644    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-fsk4s\""
Oct 30 13:03:02.029805 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.029737    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 13:03:02.029805 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.029784    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-tzhgp\""
Oct 30 13:03:02.029887 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.029828    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 13:03:02.029935 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.029900    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-k8kgp\""
Oct 30 13:03:02.030537 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.030520    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 13:03:02.030667 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.030652    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 13:03:02.030713 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.030655    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 13:03:02.031643 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.031621    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 13:03:02.031729 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.031633    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 13:03:02.031777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.031754    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 13:03:02.031809 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.031799    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 13:03:02.032014 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.031996    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 13:03:02.032070 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.032047    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 13:03:02.032320 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.032279    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 13:03:02.032320 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.032310    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 13:03:02.032320 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.032324    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 13:03:02.032520 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.032313    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 13:03:02.042606 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.042587    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 13:03:02.042606 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.042588    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 13:03:02.042749 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.042617    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 13:03:02.042749 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.042672    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 13:03:02.042749 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.042675    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 13:03:02.042749 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.042690    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 13:03:02.042930 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.042772    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 13:03:02.042930 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.042882    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 13:03:02.042930 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.042916    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 13:03:02.042930 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.042927    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 13:03:02.043264 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.042934    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 13:03:02.043264 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.042959    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 13:03:02.043264 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.042978    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 13:03:02.043264 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.042988    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 13:03:02.043264 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.043008    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 13:03:02.043264 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.043019    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 13:03:02.043264 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.043030    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 13:03:02.043264 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.043049    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 13:03:02.043264 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.043066    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 13:03:02.043264 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.043066    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 13:03:02.043264 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.043068    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 13:03:02.043264 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.043075    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 13:03:02.043264 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.043098    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 13:03:02.043264 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.043083    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 13:03:02.043264 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.043109    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 13:03:02.043264 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.043103    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 13:03:02.043264 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.043150    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 13:03:02.043264 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.043167    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 13:03:02.043264 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.043122    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 13:03:02.043264 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.043180    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 13:03:02.043264 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.043193    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 13:03:02.043264 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.043185    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 13:03:02.043264 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.043155    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 13:03:02.044192 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.043514    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 13:03:02.044192 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.043740    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 13:03:02.044192 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.043798    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 13:03:02.045570 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.045546    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 13:03:02.052462 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.052445    5093 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 13:03:02.064508 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.064489    5093 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-31.ec2.internal"
Oct 30 13:03:02.064589 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.064575    5093 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-31.ec2.internal"
Oct 30 13:03:02.088003 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.087983    5093 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 13:03:02.088003 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.088000    5093 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 13:03:02.088122 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:02.088024    5093 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-31.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-31.ec2.internal"
Oct 30 13:03:02.088122 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:02.088043    5093 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-31.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-31.ec2.internal"
Oct 30 13:03:02.110358 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.110332    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-host-var-lib-cni-multus\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:03:02.110358 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.110360    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-registry-tls\") pod \"image-registry-5b97bd7c47-dssmx\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") " pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:03:02.110491 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.110381    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz5ls\" (UniqueName: \"kubernetes.io/projected/a07f5a09-920e-462f-ab6d-7a2ac53d199e-kube-api-access-gz5ls\") pod \"network-check-target-rxd99\" (UID: \"a07f5a09-920e-462f-ab6d-7a2ac53d199e\") " pod="openshift-network-diagnostics/network-check-target-rxd99"
Oct 30 13:03:02.110491 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.110398    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-registry-tls\") pod \"image-registry-9677cf66b-kpdfl\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") " pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:03:02.110491 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.110466    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lkrg\" (UniqueName: \"kubernetes.io/projected/f93ee017-ab79-447c-90e0-5b85b71dba24-kube-api-access-2lkrg\") pod \"console-operator-7cd56b8dbc-6jdj4\" (UID: \"f93ee017-ab79-447c-90e0-5b85b71dba24\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6jdj4"
Oct 30 13:03:02.110612 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.110525    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48dsv\" (UniqueName: \"kubernetes.io/projected/5cbda802-f819-4ea6-846e-440cea4afed2-kube-api-access-48dsv\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lt2qg\" (UID: \"5cbda802-f819-4ea6-846e-440cea4afed2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lt2qg"
Oct 30 13:03:02.110612 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.110570    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5a16e7b2-e45d-480a-a7bc-d914640b9867-hosts-file\") pod \"node-resolver-gltng\" (UID: \"5a16e7b2-e45d-480a-a7bc-d914640b9867\") " pod="openshift-dns/node-resolver-gltng"
Oct 30 13:03:02.110612 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.110591    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9m9z\" (UniqueName: \"kubernetes.io/projected/5a16e7b2-e45d-480a-a7bc-d914640b9867-kube-api-access-z9m9z\") pod \"node-resolver-gltng\" (UID: \"5a16e7b2-e45d-480a-a7bc-d914640b9867\") " pod="openshift-dns/node-resolver-gltng"
Oct 30 13:03:02.110612 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.110607    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-metrics-certs\") pod \"router-default-668b54766c-wgk28\" (UID: \"d5db3ba8-f1de-4fa5-9839-e7e8200da58f\") " pod="openshift-ingress/router-default-668b54766c-wgk28"
Oct 30 13:03:02.110783 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.110630    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whwnz\" (UniqueName: \"kubernetes.io/projected/a323e312-dd91-486f-985d-92b9ab8cca04-kube-api-access-whwnz\") pod \"aws-ebs-csi-driver-node-9q945\" (UID: \"a323e312-dd91-486f-985d-92b9ab8cca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q945"
Oct 30 13:03:02.110783 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.110704    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/183f23e7-ae80-445b-960c-fff9b410ba5b-cni-binary-copy\") pod \"multus-additional-cni-plugins-lrqnm\" (UID: \"183f23e7-ae80-445b-960c-fff9b410ba5b\") " pod="openshift-multus/multus-additional-cni-plugins-lrqnm"
Oct 30 13:03:02.111196 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.110792    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0d0b2655-d52d-4c95-9589-d45237fa1057-config-volume\") pod \"dns-default-8nb7s\" (UID: \"0d0b2655-d52d-4c95-9589-d45237fa1057\") " pod="openshift-dns/dns-default-8nb7s"
Oct 30 13:03:02.111196 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.110824    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f1fb4abc-b161-461c-b8ef-bd1212010510-etc-tuned\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:03:02.111196 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.110855    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-host-var-lib-kubelet\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:03:02.111196 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.110877    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-hostroot\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:03:02.111196 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.110910    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-multus-conf-dir\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:03:02.111196 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.110922    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f1fb4abc-b161-461c-b8ef-bd1212010510-etc-tuned\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:03:02.111196 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.110954    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-etc-openvswitch\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.111196 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.111004    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nrlb\" (UniqueName: \"kubernetes.io/projected/79664154-5454-4294-baad-12bdfad6fe68-kube-api-access-7nrlb\") pod \"service-ca-operator-544d749c8c-fhnzt\" (UID: \"79664154-5454-4294-baad-12bdfad6fe68\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fhnzt"
Oct 30 13:03:02.111196 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.111033    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ceb0bb1-def2-44a5-9e31-ceffd848125d-data-volume\") pod \"insights-runtime-extractor-nk2zz\" (UID: \"7ceb0bb1-def2-44a5-9e31-ceffd848125d\") " pod="openshift-insights/insights-runtime-extractor-nk2zz"
Oct 30 13:03:02.111196 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.111051    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0d0b2655-d52d-4c95-9589-d45237fa1057-config-volume\") pod \"dns-default-8nb7s\" (UID: \"0d0b2655-d52d-4c95-9589-d45237fa1057\") " pod="openshift-dns/dns-default-8nb7s"
Oct 30 13:03:02.111196 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.111058    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/183f23e7-ae80-445b-960c-fff9b410ba5b-cni-binary-copy\") pod \"multus-additional-cni-plugins-lrqnm\" (UID: \"183f23e7-ae80-445b-960c-fff9b410ba5b\") " pod="openshift-multus/multus-additional-cni-plugins-lrqnm"
Oct 30 13:03:02.111196 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.111053    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-etc-kubernetes\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:03:02.111196 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.111119    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ceb0bb1-def2-44a5-9e31-ceffd848125d-data-volume\") pod \"insights-runtime-extractor-nk2zz\" (UID: \"7ceb0bb1-def2-44a5-9e31-ceffd848125d\") " pod="openshift-insights/insights-runtime-extractor-nk2zz"
Oct 30 13:03:02.111196 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.111124    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-default-certificate\") pod \"router-default-668b54766c-wgk28\" (UID: \"d5db3ba8-f1de-4fa5-9839-e7e8200da58f\") " pod="openshift-ingress/router-default-668b54766c-wgk28"
Oct 30 13:03:02.111196 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.111187    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-cnibin\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:03:02.111196 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.111207    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/beb2b7e4-b43d-42fc-9e8a-ecdfa6994205-agent-certs\") pod \"konnectivity-agent-wpqvv\" (UID: \"beb2b7e4-b43d-42fc-9e8a-ecdfa6994205\") " pod="kube-system/konnectivity-agent-wpqvv"
Oct 30 13:03:02.111799 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.111222    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-run-systemd\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.111799 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.111241    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdh8w\" (UniqueName: \"kubernetes.io/projected/f4acf40a-d5b0-46e2-b87f-3b96fc9678c4-kube-api-access-xdh8w\") pod \"periodic-gathering-9wlmq-z26dn\" (UID: \"f4acf40a-d5b0-46e2-b87f-3b96fc9678c4\") " pod="openshift-insights/periodic-gathering-9wlmq-z26dn"
Oct 30 13:03:02.111799 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.111264    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/10b774a1-1476-4f91-b6e5-d52b3f45700e-original-pull-secret\") pod \"global-pull-secret-syncer-wrfqz\" (UID: \"10b774a1-1476-4f91-b6e5-d52b3f45700e\") " pod="kube-system/global-pull-secret-syncer-wrfqz"
Oct 30 13:03:02.111799 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.111378    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-bound-sa-token\") pod \"image-registry-5b97bd7c47-dssmx\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") " pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:03:02.111799 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.111411    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnx68\" (UniqueName: \"kubernetes.io/projected/4ad66a0c-8dc0-40a4-a1b2-28d26b0de94b-kube-api-access-rnx68\") pod \"network-check-source-6f78d59f99-r7vrw\" (UID: \"4ad66a0c-8dc0-40a4-a1b2-28d26b0de94b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-r7vrw"
Oct 30 13:03:02.111799 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.111441    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gdf9\" (UniqueName: \"kubernetes.io/projected/087058f5-e286-4ac7-8ed8-a280da332878-kube-api-access-6gdf9\") pod \"node-ca-f7mvz\" (UID: \"087058f5-e286-4ac7-8ed8-a280da332878\") " pod="openshift-image-registry/node-ca-f7mvz"
Oct 30 13:03:02.111799 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.111563    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-image-registry-private-configuration\") pod \"image-registry-9677cf66b-kpdfl\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") " pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:03:02.111799 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.111589    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7ceb0bb1-def2-44a5-9e31-ceffd848125d-crio-socket\") pod \"insights-runtime-extractor-nk2zz\" (UID: \"7ceb0bb1-def2-44a5-9e31-ceffd848125d\") " pod="openshift-insights/insights-runtime-extractor-nk2zz"
Oct 30 13:03:02.111799 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.111609    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f4acf40a-d5b0-46e2-b87f-3b96fc9678c4-archives-path\") pod \"periodic-gathering-9wlmq-z26dn\" (UID: \"f4acf40a-d5b0-46e2-b87f-3b96fc9678c4\") " pod="openshift-insights/periodic-gathering-9wlmq-z26dn"
Oct 30 13:03:02.111799 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.111635    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4acf40a-d5b0-46e2-b87f-3b96fc9678c4-service-ca-bundle\") pod \"periodic-gathering-9wlmq-z26dn\" (UID: \"f4acf40a-d5b0-46e2-b87f-3b96fc9678c4\") " pod="openshift-insights/periodic-gathering-9wlmq-z26dn"
Oct 30 13:03:02.111799 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.111700    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6f7b\" (UniqueName: \"kubernetes.io/projected/de69ec51-fd3a-4bc4-ac17-ff02bc582905-kube-api-access-v6f7b\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:03:02.111799 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.111747    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-run-ovn\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.111799 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.111775    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/79664154-5454-4294-baad-12bdfad6fe68-serving-cert\") pod \"service-ca-operator-544d749c8c-fhnzt\" (UID: \"79664154-5454-4294-baad-12bdfad6fe68\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fhnzt"
Oct 30 13:03:02.111799 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.111787    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f4acf40a-d5b0-46e2-b87f-3b96fc9678c4-archives-path\") pod \"periodic-gathering-9wlmq-z26dn\" (UID: \"f4acf40a-d5b0-46e2-b87f-3b96fc9678c4\") " pod="openshift-insights/periodic-gathering-9wlmq-z26dn"
Oct 30 13:03:02.112224 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.111821    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a16e7b2-e45d-480a-a7bc-d914640b9867-tmp-dir\") pod \"node-resolver-gltng\" (UID: \"5a16e7b2-e45d-480a-a7bc-d914640b9867\") " pod="openshift-dns/node-resolver-gltng"
Oct 30 13:03:02.112224 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.111849    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r598w\" (UniqueName: \"kubernetes.io/projected/e640effb-1ff4-40f2-8d7b-53732edcd2bf-kube-api-access-r598w\") pod \"cluster-monitoring-operator-884576c57-x8wqt\" (UID: \"e640effb-1ff4-40f2-8d7b-53732edcd2bf\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x8wqt"
Oct 30 13:03:02.112224 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.111873    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-system-cni-dir\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:03:02.112224 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.111890    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-host-run-netns\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:03:02.112224 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.111899    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a16e7b2-e45d-480a-a7bc-d914640b9867-tmp-dir\") pod \"node-resolver-gltng\" (UID: \"5a16e7b2-e45d-480a-a7bc-d914640b9867\") " pod="openshift-dns/node-resolver-gltng"
Oct 30 13:03:02.112224 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.111908    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/feda4324-4a5f-491b-a04c-202c222261a0-registry-certificates\") pod \"image-registry-5b97bd7c47-dssmx\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") " pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:03:02.112224 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.111953    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/155bbdd4-b566-467f-88c6-ec5b4b881a09-service-ca-bundle\") pod \"insights-operator-7c4986d449-lgssx\" (UID: \"155bbdd4-b566-467f-88c6-ec5b4b881a09\") " pod="openshift-insights/insights-operator-7c4986d449-lgssx"
Oct 30 13:03:02.112224 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.111979    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/63086243-e82d-4fc5-bfef-e4241c22ad4e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2w8jt\" (UID: \"63086243-e82d-4fc5-bfef-e4241c22ad4e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2w8jt"
Oct 30 13:03:02.112224 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.111980    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4acf40a-d5b0-46e2-b87f-3b96fc9678c4-service-ca-bundle\") pod \"periodic-gathering-9wlmq-z26dn\" (UID: \"f4acf40a-d5b0-46e2-b87f-3b96fc9678c4\") " pod="openshift-insights/periodic-gathering-9wlmq-z26dn"
Oct 30 13:03:02.112224 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.111998    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/183f23e7-ae80-445b-960c-fff9b410ba5b-system-cni-dir\") pod \"multus-additional-cni-plugins-lrqnm\" (UID: \"183f23e7-ae80-445b-960c-fff9b410ba5b\") " pod="openshift-multus/multus-additional-cni-plugins-lrqnm"
Oct 30 13:03:02.112224 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.112106    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/155bbdd4-b566-467f-88c6-ec5b4b881a09-service-ca-bundle\") pod \"insights-operator-7c4986d449-lgssx\" (UID: \"155bbdd4-b566-467f-88c6-ec5b4b881a09\") " pod="openshift-insights/insights-operator-7c4986d449-lgssx"
Oct 30 13:03:02.112224 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.112122    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/183f23e7-ae80-445b-960c-fff9b410ba5b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lrqnm\" (UID: \"183f23e7-ae80-445b-960c-fff9b410ba5b\") " pod="openshift-multus/multus-additional-cni-plugins-lrqnm"
Oct 30 13:03:02.112224 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.112175    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfxrl\" (UniqueName: \"kubernetes.io/projected/abf08e3d-eec3-4f73-92a6-4ffb691cbd2d-kube-api-access-tfxrl\") pod \"volume-data-source-validator-58c9f5f46d-lvf7h\" (UID: \"abf08e3d-eec3-4f73-92a6-4ffb691cbd2d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-lvf7h"
Oct 30 13:03:02.112224 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.112202    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-etc-sysctl-d\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:03:02.112224 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.112229    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-262nq\" (UniqueName: \"kubernetes.io/projected/8021bc0b-4447-4f04-b9bd-5f80e9cf3507-kube-api-access-262nq\") pod \"kas-connection-checker-685895c7c9-rbf7t\" (UID: \"8021bc0b-4447-4f04-b9bd-5f80e9cf3507\") " pod="kube-system/kas-connection-checker-685895c7c9-rbf7t"
Oct 30 13:03:02.112782 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.112304    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/183f23e7-ae80-445b-960c-fff9b410ba5b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lrqnm\" (UID: \"183f23e7-ae80-445b-960c-fff9b410ba5b\") " pod="openshift-multus/multus-additional-cni-plugins-lrqnm"
Oct 30 13:03:02.112782 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.112336    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f93ee017-ab79-447c-90e0-5b85b71dba24-serving-cert\") pod \"console-operator-7cd56b8dbc-6jdj4\" (UID: \"f93ee017-ab79-447c-90e0-5b85b71dba24\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6jdj4"
Oct 30 13:03:02.112782 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.112379    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-node-log\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.112782 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.112411    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-host-cni-bin\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.112782 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.112441    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-ovn-node-metrics-cert\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.112782 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.112481    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f93ee017-ab79-447c-90e0-5b85b71dba24-trusted-ca\") pod \"console-operator-7cd56b8dbc-6jdj4\" (UID: \"f93ee017-ab79-447c-90e0-5b85b71dba24\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6jdj4"
Oct 30 13:03:02.112782 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.112581    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d0b2655-d52d-4c95-9589-d45237fa1057-tmp-dir\") pod \"dns-default-8nb7s\" (UID: \"0d0b2655-d52d-4c95-9589-d45237fa1057\") " pod="openshift-dns/dns-default-8nb7s"
Oct 30 13:03:02.112782 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.112613    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-etc-sysctl-conf\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:03:02.112782 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.112642    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e640effb-1ff4-40f2-8d7b-53732edcd2bf-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-x8wqt\" (UID: \"e640effb-1ff4-40f2-8d7b-53732edcd2bf\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x8wqt"
Oct 30 13:03:02.112782 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.112665    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d0b2655-d52d-4c95-9589-d45237fa1057-tmp-dir\") pod \"dns-default-8nb7s\" (UID: \"0d0b2655-d52d-4c95-9589-d45237fa1057\") " pod="openshift-dns/dns-default-8nb7s"
Oct 30 13:03:02.112782 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.112670    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-var-lib-openvswitch\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.112782 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.112694    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/183f23e7-ae80-445b-960c-fff9b410ba5b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lrqnm\" (UID: \"183f23e7-ae80-445b-960c-fff9b410ba5b\") " pod="openshift-multus/multus-additional-cni-plugins-lrqnm"
Oct 30 13:03:02.112782 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.112770    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/10b774a1-1476-4f91-b6e5-d52b3f45700e-kubelet-config\") pod \"global-pull-secret-syncer-wrfqz\" (UID: \"10b774a1-1476-4f91-b6e5-d52b3f45700e\") " pod="kube-system/global-pull-secret-syncer-wrfqz"
Oct 30 13:03:02.112782 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.112784    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/feda4324-4a5f-491b-a04c-202c222261a0-registry-certificates\") pod \"image-registry-5b97bd7c47-dssmx\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") " pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:03:02.113205 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.112797    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-sys\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:03:02.113205 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.112841    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kncj9\" (UniqueName: \"kubernetes.io/projected/7cca8a39-bee0-4705-bcde-b31060e44944-kube-api-access-kncj9\") pod \"cluster-samples-operator-6ddb6c8b94-gxcrm\" (UID: \"7cca8a39-bee0-4705-bcde-b31060e44944\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gxcrm"
Oct 30 13:03:02.113205 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.112848    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f93ee017-ab79-447c-90e0-5b85b71dba24-serving-cert\") pod \"console-operator-7cd56b8dbc-6jdj4\" (UID: \"f93ee017-ab79-447c-90e0-5b85b71dba24\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6jdj4"
Oct 30 13:03:02.113205 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.112866    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk568\" (UniqueName: \"kubernetes.io/projected/1a25d5a7-83d4-443b-98ae-fc7b603cfc5f-kube-api-access-qk568\") pod \"ingress-canary-wmb4b\" (UID: \"1a25d5a7-83d4-443b-98ae-fc7b603cfc5f\") " pod="openshift-ingress-canary/ingress-canary-wmb4b"
Oct 30 13:03:02.113205 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.112883    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-trusted-ca\") pod \"image-registry-9677cf66b-kpdfl\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") " pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:03:02.113205 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.112888    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f93ee017-ab79-447c-90e0-5b85b71dba24-trusted-ca\") pod \"console-operator-7cd56b8dbc-6jdj4\" (UID: \"f93ee017-ab79-447c-90e0-5b85b71dba24\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6jdj4"
Oct 30 13:03:02.113205 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.112899    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7d29458a-da6c-461c-9941-afa323061f30-signing-key\") pod \"service-ca-5fffc97bdf-x8ck2\" (UID: \"7d29458a-da6c-461c-9941-afa323061f30\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x8ck2"
Oct 30 13:03:02.113205 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.112881    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e640effb-1ff4-40f2-8d7b-53732edcd2bf-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-x8wqt\" (UID: \"e640effb-1ff4-40f2-8d7b-53732edcd2bf\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x8wqt"
Oct 30 13:03:02.113205 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.112915    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7d29458a-da6c-461c-9941-afa323061f30-signing-cabundle\") pod \"service-ca-5fffc97bdf-x8ck2\" (UID: \"7d29458a-da6c-461c-9941-afa323061f30\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x8ck2"
Oct 30 13:03:02.113205 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.112945    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-host-var-lib-cni-bin\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:03:02.113205 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.112972    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a323e312-dd91-486f-985d-92b9ab8cca04-device-dir\") pod \"aws-ebs-csi-driver-node-9q945\" (UID: \"a323e312-dd91-486f-985d-92b9ab8cca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q945"
Oct 30 13:03:02.113205 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.113002    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-registry-certificates\") pod \"image-registry-9677cf66b-kpdfl\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") " pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:03:02.113205 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.113028    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0c8e72ed-d54d-4e6a-bf66-65a16430351d-host-slash\") pod \"iptables-alerter-5swg6\" (UID: \"0c8e72ed-d54d-4e6a-bf66-65a16430351d\") " pod="openshift-network-operator/iptables-alerter-5swg6"
Oct 30 13:03:02.113205 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.113041    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7d29458a-da6c-461c-9941-afa323061f30-signing-cabundle\") pod \"service-ca-5fffc97bdf-x8ck2\" (UID: \"7d29458a-da6c-461c-9941-afa323061f30\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x8ck2"
Oct 30 13:03:02.113205 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.113116    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4acf40a-d5b0-46e2-b87f-3b96fc9678c4-trusted-ca-bundle\") pod \"periodic-gathering-9wlmq-z26dn\" (UID: \"f4acf40a-d5b0-46e2-b87f-3b96fc9678c4\") " pod="openshift-insights/periodic-gathering-9wlmq-z26dn"
Oct 30 13:03:02.113205 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.113156    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb526\" (UniqueName: \"kubernetes.io/projected/f1fb4abc-b161-461c-b8ef-bd1212010510-kube-api-access-zb526\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:03:02.113205 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.113174    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e640effb-1ff4-40f2-8d7b-53732edcd2bf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-x8wqt\" (UID: \"e640effb-1ff4-40f2-8d7b-53732edcd2bf\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x8wqt"
Oct 30 13:03:02.113205 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.113193    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/087058f5-e286-4ac7-8ed8-a280da332878-serviceca\") pod \"node-ca-f7mvz\" (UID: \"087058f5-e286-4ac7-8ed8-a280da332878\") " pod="openshift-image-registry/node-ca-f7mvz"
Oct 30 13:03:02.113205 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.113210    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjgw5\" (UniqueName: \"kubernetes.io/projected/183f23e7-ae80-445b-960c-fff9b410ba5b-kube-api-access-hjgw5\") pod \"multus-additional-cni-plugins-lrqnm\" (UID: \"183f23e7-ae80-445b-960c-fff9b410ba5b\") " pod="openshift-multus/multus-additional-cni-plugins-lrqnm"
Oct 30 13:03:02.113694 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.113233    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7ceb0bb1-def2-44a5-9e31-ceffd848125d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nk2zz\" (UID: \"7ceb0bb1-def2-44a5-9e31-ceffd848125d\") " pod="openshift-insights/insights-runtime-extractor-nk2zz"
Oct 30 13:03:02.113694 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.113249    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a32a253a-ecaf-4f90-af1c-27a7e53b9629-metrics-certs\") pod \"network-metrics-daemon-mbncl\" (UID: \"a32a253a-ecaf-4f90-af1c-27a7e53b9629\") " pod="openshift-multus/network-metrics-daemon-mbncl"
Oct 30 13:03:02.113694 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.113266    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/feda4324-4a5f-491b-a04c-202c222261a0-trusted-ca\") pod \"image-registry-5b97bd7c47-dssmx\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") " pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:03:02.113694 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.113280    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-systemd-units\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.113694 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.113302    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-env-overrides\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.113694 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.113319    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/feda4324-4a5f-491b-a04c-202c222261a0-image-registry-private-configuration\") pod \"image-registry-5b97bd7c47-dssmx\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") " pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:03:02.113694 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.113340    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f93ee017-ab79-447c-90e0-5b85b71dba24-config\") pod \"console-operator-7cd56b8dbc-6jdj4\" (UID: \"f93ee017-ab79-447c-90e0-5b85b71dba24\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6jdj4"
Oct 30 13:03:02.113694 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.113358    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/087058f5-e286-4ac7-8ed8-a280da332878-host\") pod \"node-ca-f7mvz\" (UID: \"087058f5-e286-4ac7-8ed8-a280da332878\") " pod="openshift-image-registry/node-ca-f7mvz"
Oct 30 13:03:02.113694 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.113368    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-trusted-ca\") pod \"image-registry-9677cf66b-kpdfl\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") " pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:03:02.113694 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.113378    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-log-socket\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.113694 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.113398    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/63086243-e82d-4fc5-bfef-e4241c22ad4e-nginx-conf\") pod \"networking-console-plugin-86c478bb88-2w8jt\" (UID: \"63086243-e82d-4fc5-bfef-e4241c22ad4e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2w8jt"
Oct 30 13:03:02.113694 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.113415    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/155bbdd4-b566-467f-88c6-ec5b4b881a09-serving-cert\") pod \"insights-operator-7c4986d449-lgssx\" (UID: \"155bbdd4-b566-467f-88c6-ec5b4b881a09\") " pod="openshift-insights/insights-operator-7c4986d449-lgssx"
Oct 30 13:03:02.113694 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.113594    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-env-overrides\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.113694 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.113596    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4acf40a-d5b0-46e2-b87f-3b96fc9678c4-trusted-ca-bundle\") pod \"periodic-gathering-9wlmq-z26dn\" (UID: \"f4acf40a-d5b0-46e2-b87f-3b96fc9678c4\") " pod="openshift-insights/periodic-gathering-9wlmq-z26dn"
Oct 30 13:03:02.114097 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.113746    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/feda4324-4a5f-491b-a04c-202c222261a0-trusted-ca\") pod \"image-registry-5b97bd7c47-dssmx\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") " pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:03:02.114097 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.113762    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f93ee017-ab79-447c-90e0-5b85b71dba24-config\") pod \"console-operator-7cd56b8dbc-6jdj4\" (UID: \"f93ee017-ab79-447c-90e0-5b85b71dba24\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6jdj4"
Oct 30 13:03:02.114097 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.113801    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsj7z\" (UniqueName: \"kubernetes.io/projected/155bbdd4-b566-467f-88c6-ec5b4b881a09-kube-api-access-lsj7z\") pod \"insights-operator-7c4986d449-lgssx\" (UID: \"155bbdd4-b566-467f-88c6-ec5b4b881a09\") " pod="openshift-insights/insights-operator-7c4986d449-lgssx"
Oct 30 13:03:02.114097 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.113833    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plb6z\" (UniqueName: \"kubernetes.io/projected/0d0b2655-d52d-4c95-9589-d45237fa1057-kube-api-access-plb6z\") pod \"dns-default-8nb7s\" (UID: \"0d0b2655-d52d-4c95-9589-d45237fa1057\") " pod="openshift-dns/dns-default-8nb7s"
Oct 30 13:03:02.114097 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.113847    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-registry-certificates\") pod \"image-registry-9677cf66b-kpdfl\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") " pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:03:02.114097 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.113860    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-stats-auth\") pod \"router-default-668b54766c-wgk28\" (UID: \"d5db3ba8-f1de-4fa5-9839-e7e8200da58f\") " pod="openshift-ingress/router-default-668b54766c-wgk28"
Oct 30 13:03:02.114097 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.113900    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/63086243-e82d-4fc5-bfef-e4241c22ad4e-nginx-conf\") pod \"networking-console-plugin-86c478bb88-2w8jt\" (UID: \"63086243-e82d-4fc5-bfef-e4241c22ad4e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2w8jt"
Oct 30 13:03:02.114097 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.113922    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-multus-cni-dir\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:03:02.114097 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.113958    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-lib-modules\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:03:02.114097 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.113982    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-host-run-multus-certs\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:03:02.114097 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.114030    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7ceb0bb1-def2-44a5-9e31-ceffd848125d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nk2zz\" (UID: \"7ceb0bb1-def2-44a5-9e31-ceffd848125d\") " pod="openshift-insights/insights-runtime-extractor-nk2zz"
Oct 30 13:03:02.114578 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.114166    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5k78\" (UniqueName: \"kubernetes.io/projected/7ceb0bb1-def2-44a5-9e31-ceffd848125d-kube-api-access-f5k78\") pod \"insights-runtime-extractor-nk2zz\" (UID: \"7ceb0bb1-def2-44a5-9e31-ceffd848125d\") " pod="openshift-insights/insights-runtime-extractor-nk2zz"
Oct 30 13:03:02.114578 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.114227    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/feda4324-4a5f-491b-a04c-202c222261a0-ca-trust-extracted\") pod \"image-registry-5b97bd7c47-dssmx\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") " pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:03:02.114578 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.114265    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7ceb0bb1-def2-44a5-9e31-ceffd848125d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nk2zz\" (UID: \"7ceb0bb1-def2-44a5-9e31-ceffd848125d\") " pod="openshift-insights/insights-runtime-extractor-nk2zz"
Oct 30 13:03:02.114578 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.114274    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbpg6\" (UniqueName: \"kubernetes.io/projected/46bedd0c-7237-4128-be98-69e4474bb535-kube-api-access-kbpg6\") pod \"kas-connection-checker-685895c7c9-22fm4\" (UID: \"46bedd0c-7237-4128-be98-69e4474bb535\") " pod="kube-system/kas-connection-checker-685895c7c9-22fm4"
Oct 30 13:03:02.114578 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.114320    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-installation-pull-secrets\") pod \"image-registry-9677cf66b-kpdfl\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") " pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:03:02.114578 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.114328    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/087058f5-e286-4ac7-8ed8-a280da332878-serviceca\") pod \"node-ca-f7mvz\" (UID: \"087058f5-e286-4ac7-8ed8-a280da332878\") " pod="openshift-image-registry/node-ca-f7mvz"
Oct 30 13:03:02.114578 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.114339    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/feda4324-4a5f-491b-a04c-202c222261a0-ca-trust-extracted\") pod \"image-registry-5b97bd7c47-dssmx\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") " pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:03:02.114578 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.114351    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdrpm\" (UniqueName: \"kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-kube-api-access-vdrpm\") pod \"image-registry-9677cf66b-kpdfl\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") " pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:03:02.114578 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.114383    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgqhj\" (UniqueName: \"kubernetes.io/projected/0c8e72ed-d54d-4e6a-bf66-65a16430351d-kube-api-access-dgqhj\") pod \"iptables-alerter-5swg6\" (UID: \"0c8e72ed-d54d-4e6a-bf66-65a16430351d\") " pod="openshift-network-operator/iptables-alerter-5swg6"
Oct 30 13:03:02.114578 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.114413    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-multus-socket-dir-parent\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:03:02.114578 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.114443    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncqh7\" (UniqueName: \"kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-kube-api-access-ncqh7\") pod \"image-registry-5b97bd7c47-dssmx\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") " pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:03:02.114578 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.114474    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-host-cni-netd\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.114578 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.114540    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/155bbdd4-b566-467f-88c6-ec5b4b881a09-snapshots\") pod \"insights-operator-7c4986d449-lgssx\" (UID: \"155bbdd4-b566-467f-88c6-ec5b4b881a09\") " pod="openshift-insights/insights-operator-7c4986d449-lgssx"
Oct 30 13:03:02.114578 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.114580    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-789vz\" (UniqueName: \"kubernetes.io/projected/7d29458a-da6c-461c-9941-afa323061f30-kube-api-access-789vz\") pod \"service-ca-5fffc97bdf-x8ck2\" (UID: \"7d29458a-da6c-461c-9941-afa323061f30\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x8ck2"
Oct 30 13:03:02.115223 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.114599    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/10b774a1-1476-4f91-b6e5-d52b3f45700e-dbus\") pod \"global-pull-secret-syncer-wrfqz\" (UID: \"10b774a1-1476-4f91-b6e5-d52b3f45700e\") " pod="kube-system/global-pull-secret-syncer-wrfqz"
Oct 30 13:03:02.115223 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.114637    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/155bbdd4-b566-467f-88c6-ec5b4b881a09-snapshots\") pod \"insights-operator-7c4986d449-lgssx\" (UID: \"155bbdd4-b566-467f-88c6-ec5b4b881a09\") " pod="openshift-insights/insights-operator-7c4986d449-lgssx"
Oct 30 13:03:02.115223 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.114730    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1a25d5a7-83d4-443b-98ae-fc7b603cfc5f-cert\") pod \"ingress-canary-wmb4b\" (UID: \"1a25d5a7-83d4-443b-98ae-fc7b603cfc5f\") " pod="openshift-ingress-canary/ingress-canary-wmb4b"
Oct 30 13:03:02.115223 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.114761    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a323e312-dd91-486f-985d-92b9ab8cca04-sys-fs\") pod \"aws-ebs-csi-driver-node-9q945\" (UID: \"a323e312-dd91-486f-985d-92b9ab8cca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q945"
Oct 30 13:03:02.115223 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.114788    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-etc-sysconfig\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:03:02.115223 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.114812    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-var-lib-kubelet\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:03:02.115223 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.114845    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/de69ec51-fd3a-4bc4-ac17-ff02bc582905-multus-daemon-config\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:03:02.115223 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.114870    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a323e312-dd91-486f-985d-92b9ab8cca04-registration-dir\") pod \"aws-ebs-csi-driver-node-9q945\" (UID: \"a323e312-dd91-486f-985d-92b9ab8cca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q945"
Oct 30 13:03:02.115223 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.114901    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/155bbdd4-b566-467f-88c6-ec5b4b881a09-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-lgssx\" (UID: \"155bbdd4-b566-467f-88c6-ec5b4b881a09\") " pod="openshift-insights/insights-operator-7c4986d449-lgssx"
Oct 30 13:03:02.115223 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.114931    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5cbda802-f819-4ea6-846e-440cea4afed2-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lt2qg\" (UID: \"5cbda802-f819-4ea6-846e-440cea4afed2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lt2qg"
Oct 30 13:03:02.115223 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.114956    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-host\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:03:02.115223 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.114980    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-etc-kubernetes\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:03:02.115223 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.115009    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/feda4324-4a5f-491b-a04c-202c222261a0-installation-pull-secrets\") pod \"image-registry-5b97bd7c47-dssmx\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") " pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:03:02.115223 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.115037    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-ovnkube-script-lib\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.115223 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.115055    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5cbda802-f819-4ea6-846e-440cea4afed2-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lt2qg\" (UID: \"5cbda802-f819-4ea6-846e-440cea4afed2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lt2qg"
Oct 30 13:03:02.115223 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.115119    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/155bbdd4-b566-467f-88c6-ec5b4b881a09-tmp\") pod \"insights-operator-7c4986d449-lgssx\" (UID: \"155bbdd4-b566-467f-88c6-ec5b4b881a09\") " pod="openshift-insights/insights-operator-7c4986d449-lgssx"
Oct 30 13:03:02.115223 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.115172    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsfql\" (UniqueName: \"kubernetes.io/projected/a32a253a-ecaf-4f90-af1c-27a7e53b9629-kube-api-access-qsfql\") pod \"network-metrics-daemon-mbncl\" (UID: \"a32a253a-ecaf-4f90-af1c-27a7e53b9629\") " pod="openshift-multus/network-metrics-daemon-mbncl"
Oct 30 13:03:02.115223 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.115193    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-service-ca-bundle\") pod \"router-default-668b54766c-wgk28\" (UID: \"d5db3ba8-f1de-4fa5-9839-e7e8200da58f\") " pod="openshift-ingress/router-default-668b54766c-wgk28"
Oct 30 13:03:02.115223 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.115204    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/155bbdd4-b566-467f-88c6-ec5b4b881a09-tmp\") pod \"insights-operator-7c4986d449-lgssx\" (UID: \"155bbdd4-b566-467f-88c6-ec5b4b881a09\") " pod="openshift-insights/insights-operator-7c4986d449-lgssx"
Oct 30 13:03:02.115223 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.115116    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/de69ec51-fd3a-4bc4-ac17-ff02bc582905-multus-daemon-config\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:03:02.115965 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.115209    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/de69ec51-fd3a-4bc4-ac17-ff02bc582905-cni-binary-copy\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:03:02.115965 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.115255    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zbfrs\""
Oct 30 13:03:02.115965 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.115298    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-ovnkube-script-lib\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.115965 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.115325    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/155bbdd4-b566-467f-88c6-ec5b4b881a09-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-lgssx\" (UID: \"155bbdd4-b566-467f-88c6-ec5b4b881a09\") " pod="openshift-insights/insights-operator-7c4986d449-lgssx"
Oct 30 13:03:02.115965 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.115329    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-host-run-k8s-cni-cncf-io\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:03:02.115965 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.115358    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0c8e72ed-d54d-4e6a-bf66-65a16430351d-iptables-alerter-script\") pod \"iptables-alerter-5swg6\" (UID: \"0c8e72ed-d54d-4e6a-bf66-65a16430351d\") " pod="openshift-network-operator/iptables-alerter-5swg6"
Oct 30 13:03:02.115965 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.115402    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/de69ec51-fd3a-4bc4-ac17-ff02bc582905-cni-binary-copy\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:03:02.115965 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.115432    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5cbda802-f819-4ea6-846e-440cea4afed2-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lt2qg\" (UID: \"5cbda802-f819-4ea6-846e-440cea4afed2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lt2qg"
Oct 30 13:03:02.115965 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.115496    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-bound-sa-token\") pod \"image-registry-9677cf66b-kpdfl\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") " pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:03:02.115965 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.115527    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/beb2b7e4-b43d-42fc-9e8a-ecdfa6994205-konnectivity-ca\") pod \"konnectivity-agent-wpqvv\" (UID: \"beb2b7e4-b43d-42fc-9e8a-ecdfa6994205\") " pod="kube-system/konnectivity-agent-wpqvv"
Oct 30 13:03:02.115965 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.115627    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0c8e72ed-d54d-4e6a-bf66-65a16430351d-iptables-alerter-script\") pod \"iptables-alerter-5swg6\" (UID: \"0c8e72ed-d54d-4e6a-bf66-65a16430351d\") " pod="openshift-network-operator/iptables-alerter-5swg6"
Oct 30 13:03:02.115965 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.115663    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-host-slash\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.115965 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.115691    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-run-openvswitch\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.115965 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.115688    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5cbda802-f819-4ea6-846e-440cea4afed2-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lt2qg\" (UID: \"5cbda802-f819-4ea6-846e-440cea4afed2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lt2qg"
Oct 30 13:03:02.115965 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.115737    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xws2p\" (UniqueName: \"kubernetes.io/projected/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-kube-api-access-xws2p\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.115965 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.115835    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/beb2b7e4-b43d-42fc-9e8a-ecdfa6994205-konnectivity-ca\") pod \"konnectivity-agent-wpqvv\" (UID: \"beb2b7e4-b43d-42fc-9e8a-ecdfa6994205\") " pod="kube-system/konnectivity-agent-wpqvv"
Oct 30 13:03:02.115965 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.115861    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/79664154-5454-4294-baad-12bdfad6fe68-config\") pod \"service-ca-operator-544d749c8c-fhnzt\" (UID: \"79664154-5454-4294-baad-12bdfad6fe68\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fhnzt"
Oct 30 13:03:02.115965 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.115878    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/183f23e7-ae80-445b-960c-fff9b410ba5b-cnibin\") pod \"multus-additional-cni-plugins-lrqnm\" (UID: \"183f23e7-ae80-445b-960c-fff9b410ba5b\") " pod="openshift-multus/multus-additional-cni-plugins-lrqnm"
Oct 30 13:03:02.115965 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.115895    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/183f23e7-ae80-445b-960c-fff9b410ba5b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lrqnm\" (UID: \"183f23e7-ae80-445b-960c-fff9b410ba5b\") " pod="openshift-multus/multus-additional-cni-plugins-lrqnm"
Oct 30 13:03:02.115965 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.115911    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a323e312-dd91-486f-985d-92b9ab8cca04-etc-selinux\") pod \"aws-ebs-csi-driver-node-9q945\" (UID: \"a323e312-dd91-486f-985d-92b9ab8cca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q945"
Oct 30 13:03:02.115965 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.115938    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-host-run-netns\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.116865 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.115981    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-ovnkube-config\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.116865 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.116028    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/79664154-5454-4294-baad-12bdfad6fe68-config\") pod \"service-ca-operator-544d749c8c-fhnzt\" (UID: \"79664154-5454-4294-baad-12bdfad6fe68\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fhnzt"
Oct 30 13:03:02.116865 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.116067    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/183f23e7-ae80-445b-960c-fff9b410ba5b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lrqnm\" (UID: \"183f23e7-ae80-445b-960c-fff9b410ba5b\") " pod="openshift-multus/multus-additional-cni-plugins-lrqnm"
Oct 30 13:03:02.116865 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.116027    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-ca-trust-extracted\") pod \"image-registry-9677cf66b-kpdfl\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") " pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:03:02.116865 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.116093    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-ca-trust-extracted\") pod \"image-registry-9677cf66b-kpdfl\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") " pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:03:02.116865 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.116099    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjjr5\" (UniqueName: \"kubernetes.io/projected/44f513ab-9115-4761-aa8b-c546212ec17b-kube-api-access-vjjr5\") pod \"kas-connection-checker-685895c7c9-wnzh2\" (UID: \"44f513ab-9115-4761-aa8b-c546212ec17b\") " pod="kube-system/kas-connection-checker-685895c7c9-wnzh2"
Oct 30 13:03:02.116865 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.116224    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-ovnkube-config\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.116865 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.116305    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-etc-systemd\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:03:02.116865 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.116323    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 13:03:02.116865 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.116336    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1fb4abc-b161-461c-b8ef-bd1212010510-tmp\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:03:02.116865 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.116368    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-host-kubelet\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.116865 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.116391    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jdqx\" (UniqueName: \"kubernetes.io/projected/7bbbc7e0-1790-459b-952a-b6b7b4d55113-kube-api-access-8jdqx\") pod \"migrator-66df5d496b-r5pcv\" (UID: \"7bbbc7e0-1790-459b-952a-b6b7b4d55113\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-r5pcv"
Oct 30 13:03:02.116865 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.116408    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a323e312-dd91-486f-985d-92b9ab8cca04-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9q945\" (UID: \"a323e312-dd91-486f-985d-92b9ab8cca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q945"
Oct 30 13:03:02.116865 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.116408    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1fb4abc-b161-461c-b8ef-bd1212010510-tmp\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:03:02.116865 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.116450    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d0b2655-d52d-4c95-9589-d45237fa1057-metrics-tls\") pod \"dns-default-8nb7s\" (UID: \"0d0b2655-d52d-4c95-9589-d45237fa1057\") " pod="openshift-dns/dns-default-8nb7s"
Oct 30 13:03:02.116865 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.116470    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 13:03:02.116865 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.116557    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-prxcs\""
Oct 30 13:03:02.116865 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.116574    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-d86xq\""
Oct 30 13:03:02.116865 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.116581    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4kjx\" (UniqueName: \"kubernetes.io/projected/3c7ab011-8fd4-4d6d-b3f2-72d6af11d3e1-kube-api-access-p4kjx\") pod \"migrator-66df5d496b-k6rlj\" (UID: \"3c7ab011-8fd4-4d6d-b3f2-72d6af11d3e1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-k6rlj"
Oct 30 13:03:02.116865 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.116621    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/7cca8a39-bee0-4705-bcde-b31060e44944-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-gxcrm\" (UID: \"7cca8a39-bee0-4705-bcde-b31060e44944\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gxcrm"
Oct 30 13:03:02.116865 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.116652    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-host-run-ovn-kubernetes\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.116865 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.116680    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-etc-modprobe-d\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:03:02.116865 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.116696    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 13:03:02.116865 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.116743    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-run\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:03:02.116865 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.116785    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rflhr\" (UniqueName: \"kubernetes.io/projected/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-kube-api-access-rflhr\") pod \"router-default-668b54766c-wgk28\" (UID: \"d5db3ba8-f1de-4fa5-9839-e7e8200da58f\") " pod="openshift-ingress/router-default-668b54766c-wgk28"
Oct 30 13:03:02.116865 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.116854    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/7cca8a39-bee0-4705-bcde-b31060e44944-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-gxcrm\" (UID: \"7cca8a39-bee0-4705-bcde-b31060e44944\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gxcrm"
Oct 30 13:03:02.117777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.116913    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7cca8a39-bee0-4705-bcde-b31060e44944-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gxcrm\" (UID: \"7cca8a39-bee0-4705-bcde-b31060e44944\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gxcrm"
Oct 30 13:03:02.117777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.116960    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 13:03:02.117777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.116962    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a323e312-dd91-486f-985d-92b9ab8cca04-socket-dir\") pod \"aws-ebs-csi-driver-node-9q945\" (UID: \"a323e312-dd91-486f-985d-92b9ab8cca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q945"
Oct 30 13:03:02.117777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.117013    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.117777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.117023    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 13:03:02.117777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.117036    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 13:03:02.117777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.117069    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tzck4\""
Oct 30 13:03:02.117777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.117078    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 13:03:02.117777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.117069    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-sxrfm\""
Oct 30 13:03:02.117777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.117074    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 13:03:02.117777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.117322    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-jp6bq\""
Oct 30 13:03:02.117777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.117327    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vz5h4\""
Oct 30 13:03:02.117777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.117374    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9hcn4\""
Oct 30 13:03:02.117777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.117380    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 13:03:02.117777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.117405    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 13:03:02.117777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.117424    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-5crmk\""
Oct 30 13:03:02.117777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.117416    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 13:03:02.117777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.117438    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 13:03:02.117777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.117530    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 13:03:02.117777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.117618    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 13:03:02.117777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.117677    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 13:03:02.117777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.117728    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-r278v\""
Oct 30 13:03:02.117777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.117740    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mxh59\""
Oct 30 13:03:02.117777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.117748    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 13:03:02.117777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.117754    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fx9p2\""
Oct 30 13:03:02.117777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.117796    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8w9b7\""
Oct 30 13:03:02.118510 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.117797    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9mrzf\""
Oct 30 13:03:02.118510 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.117817    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tdfvc\""
Oct 30 13:03:02.118510 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.117810    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-76hdh\""
Oct 30 13:03:02.118510 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.117799    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-9xcbx\""
Oct 30 13:03:02.118510 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.117848    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-m8jrt\""
Oct 30 13:03:02.118510 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.117851    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rrlh9\""
Oct 30 13:03:02.118510 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.117896    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-hwzrd\""
Oct 30 13:03:02.118510 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.117927    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 13:03:02.118510 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.118008    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-w2zkq\""
Oct 30 13:03:02.118510 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.118043    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-vqv56\""
Oct 30 13:03:02.118510 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.118052    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-mc5c8\""
Oct 30 13:03:02.118510 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.118061    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-9qjrr\""
Oct 30 13:03:02.118510 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.118074    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 13:03:02.118510 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.118125    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 13:03:02.122448 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.122429    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-default-certificate\") pod \"router-default-668b54766c-wgk28\" (UID: \"d5db3ba8-f1de-4fa5-9839-e7e8200da58f\") " pod="openshift-ingress/router-default-668b54766c-wgk28"
Oct 30 13:03:02.122448 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.122444    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/10b774a1-1476-4f91-b6e5-d52b3f45700e-original-pull-secret\") pod \"global-pull-secret-syncer-wrfqz\" (UID: \"10b774a1-1476-4f91-b6e5-d52b3f45700e\") " pod="kube-system/global-pull-secret-syncer-wrfqz"
Oct 30 13:03:02.122522 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.122482    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-image-registry-private-configuration\") pod \"image-registry-9677cf66b-kpdfl\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") " pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:03:02.122552 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.122540    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/79664154-5454-4294-baad-12bdfad6fe68-serving-cert\") pod \"service-ca-operator-544d749c8c-fhnzt\" (UID: \"79664154-5454-4294-baad-12bdfad6fe68\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fhnzt"
Oct 30 13:03:02.122584 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.122566    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/beb2b7e4-b43d-42fc-9e8a-ecdfa6994205-agent-certs\") pod \"konnectivity-agent-wpqvv\" (UID: \"beb2b7e4-b43d-42fc-9e8a-ecdfa6994205\") " pod="kube-system/konnectivity-agent-wpqvv"
Oct 30 13:03:02.122638 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.122626    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-ovn-node-metrics-cert\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.123643 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.123627    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/155bbdd4-b566-467f-88c6-ec5b4b881a09-serving-cert\") pod \"insights-operator-7c4986d449-lgssx\" (UID: \"155bbdd4-b566-467f-88c6-ec5b4b881a09\") " pod="openshift-insights/insights-operator-7c4986d449-lgssx"
Oct 30 13:03:02.123753 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.123738    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7d29458a-da6c-461c-9941-afa323061f30-signing-key\") pod \"service-ca-5fffc97bdf-x8ck2\" (UID: \"7d29458a-da6c-461c-9941-afa323061f30\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x8ck2"
Oct 30 13:03:02.123809 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.123795    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/feda4324-4a5f-491b-a04c-202c222261a0-image-registry-private-configuration\") pod \"image-registry-5b97bd7c47-dssmx\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") " pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:03:02.124826 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.124814    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-stats-auth\") pod \"router-default-668b54766c-wgk28\" (UID: \"d5db3ba8-f1de-4fa5-9839-e7e8200da58f\") " pod="openshift-ingress/router-default-668b54766c-wgk28"
Oct 30 13:03:02.124865 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.124814    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-installation-pull-secrets\") pod \"image-registry-9677cf66b-kpdfl\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") " pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:03:02.124934 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.124921    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1a25d5a7-83d4-443b-98ae-fc7b603cfc5f-cert\") pod \"ingress-canary-wmb4b\" (UID: \"1a25d5a7-83d4-443b-98ae-fc7b603cfc5f\") " pod="openshift-ingress-canary/ingress-canary-wmb4b"
Oct 30 13:03:02.125979 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.125964    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/feda4324-4a5f-491b-a04c-202c222261a0-installation-pull-secrets\") pod \"image-registry-5b97bd7c47-dssmx\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") " pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:03:02.132042 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.132017    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nrlb\" (UniqueName: \"kubernetes.io/projected/79664154-5454-4294-baad-12bdfad6fe68-kube-api-access-7nrlb\") pod \"service-ca-operator-544d749c8c-fhnzt\" (UID: \"79664154-5454-4294-baad-12bdfad6fe68\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fhnzt"
Oct 30 13:03:02.132251 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.132230    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz5ls\" (UniqueName: \"kubernetes.io/projected/a07f5a09-920e-462f-ab6d-7a2ac53d199e-kube-api-access-gz5ls\") pod \"network-check-target-rxd99\" (UID: \"a07f5a09-920e-462f-ab6d-7a2ac53d199e\") " pod="openshift-network-diagnostics/network-check-target-rxd99"
Oct 30 13:03:02.132353 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.132315    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whwnz\" (UniqueName: \"kubernetes.io/projected/a323e312-dd91-486f-985d-92b9ab8cca04-kube-api-access-whwnz\") pod \"aws-ebs-csi-driver-node-9q945\" (UID: \"a323e312-dd91-486f-985d-92b9ab8cca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q945"
Oct 30 13:03:02.132488 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.132466    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gdf9\" (UniqueName: \"kubernetes.io/projected/087058f5-e286-4ac7-8ed8-a280da332878-kube-api-access-6gdf9\") pod \"node-ca-f7mvz\" (UID: \"087058f5-e286-4ac7-8ed8-a280da332878\") " pod="openshift-image-registry/node-ca-f7mvz"
Oct 30 13:03:02.132559 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.132503    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9m9z\" (UniqueName: \"kubernetes.io/projected/5a16e7b2-e45d-480a-a7bc-d914640b9867-kube-api-access-z9m9z\") pod \"node-resolver-gltng\" (UID: \"5a16e7b2-e45d-480a-a7bc-d914640b9867\") " pod="openshift-dns/node-resolver-gltng"
Oct 30 13:03:02.132559 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.132510    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-bound-sa-token\") pod \"image-registry-5b97bd7c47-dssmx\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") " pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:03:02.132559 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.132546    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lkrg\" (UniqueName: \"kubernetes.io/projected/f93ee017-ab79-447c-90e0-5b85b71dba24-kube-api-access-2lkrg\") pod \"console-operator-7cd56b8dbc-6jdj4\" (UID: \"f93ee017-ab79-447c-90e0-5b85b71dba24\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6jdj4"
Oct 30 13:03:02.132782 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.132767    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnx68\" (UniqueName: \"kubernetes.io/projected/4ad66a0c-8dc0-40a4-a1b2-28d26b0de94b-kube-api-access-rnx68\") pod \"network-check-source-6f78d59f99-r7vrw\" (UID: \"4ad66a0c-8dc0-40a4-a1b2-28d26b0de94b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-r7vrw"
Oct 30 13:03:02.132814 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.132792    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r598w\" (UniqueName: \"kubernetes.io/projected/e640effb-1ff4-40f2-8d7b-53732edcd2bf-kube-api-access-r598w\") pod \"cluster-monitoring-operator-884576c57-x8wqt\" (UID: \"e640effb-1ff4-40f2-8d7b-53732edcd2bf\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x8wqt"
Oct 30 13:03:02.133663 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.133641    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48dsv\" (UniqueName: \"kubernetes.io/projected/5cbda802-f819-4ea6-846e-440cea4afed2-kube-api-access-48dsv\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lt2qg\" (UID: \"5cbda802-f819-4ea6-846e-440cea4afed2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lt2qg"
Oct 30 13:03:02.133768 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.133674    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6f7b\" (UniqueName: \"kubernetes.io/projected/de69ec51-fd3a-4bc4-ac17-ff02bc582905-kube-api-access-v6f7b\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:03:02.133923 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.133907    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdh8w\" (UniqueName: \"kubernetes.io/projected/f4acf40a-d5b0-46e2-b87f-3b96fc9678c4-kube-api-access-xdh8w\") pod \"periodic-gathering-9wlmq-z26dn\" (UID: \"f4acf40a-d5b0-46e2-b87f-3b96fc9678c4\") " pod="openshift-insights/periodic-gathering-9wlmq-z26dn"
Oct 30 13:03:02.134917 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.134899    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfxrl\" (UniqueName: \"kubernetes.io/projected/abf08e3d-eec3-4f73-92a6-4ffb691cbd2d-kube-api-access-tfxrl\") pod \"volume-data-source-validator-58c9f5f46d-lvf7h\" (UID: \"abf08e3d-eec3-4f73-92a6-4ffb691cbd2d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-lvf7h"
Oct 30 13:03:02.134973 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.134931    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-262nq\" (UniqueName: \"kubernetes.io/projected/8021bc0b-4447-4f04-b9bd-5f80e9cf3507-kube-api-access-262nq\") pod \"kas-connection-checker-685895c7c9-rbf7t\" (UID: \"8021bc0b-4447-4f04-b9bd-5f80e9cf3507\") " pod="kube-system/kas-connection-checker-685895c7c9-rbf7t"
Oct 30 13:03:02.135409 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.135386    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kncj9\" (UniqueName: \"kubernetes.io/projected/7cca8a39-bee0-4705-bcde-b31060e44944-kube-api-access-kncj9\") pod \"cluster-samples-operator-6ddb6c8b94-gxcrm\" (UID: \"7cca8a39-bee0-4705-bcde-b31060e44944\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gxcrm"
Oct 30 13:03:02.135544 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.135522    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjgw5\" (UniqueName: \"kubernetes.io/projected/183f23e7-ae80-445b-960c-fff9b410ba5b-kube-api-access-hjgw5\") pod \"multus-additional-cni-plugins-lrqnm\" (UID: \"183f23e7-ae80-445b-960c-fff9b410ba5b\") " pod="openshift-multus/multus-additional-cni-plugins-lrqnm"
Oct 30 13:03:02.135640 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.135623    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb526\" (UniqueName: \"kubernetes.io/projected/f1fb4abc-b161-461c-b8ef-bd1212010510-kube-api-access-zb526\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:03:02.135685 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.135656    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk568\" (UniqueName: \"kubernetes.io/projected/1a25d5a7-83d4-443b-98ae-fc7b603cfc5f-kube-api-access-qk568\") pod \"ingress-canary-wmb4b\" (UID: \"1a25d5a7-83d4-443b-98ae-fc7b603cfc5f\") " pod="openshift-ingress-canary/ingress-canary-wmb4b"
Oct 30 13:03:02.137671 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.137653    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plb6z\" (UniqueName: \"kubernetes.io/projected/0d0b2655-d52d-4c95-9589-d45237fa1057-kube-api-access-plb6z\") pod \"dns-default-8nb7s\" (UID: \"0d0b2655-d52d-4c95-9589-d45237fa1057\") " pod="openshift-dns/dns-default-8nb7s"
Oct 30 13:03:02.137957 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.137931    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5k78\" (UniqueName: \"kubernetes.io/projected/7ceb0bb1-def2-44a5-9e31-ceffd848125d-kube-api-access-f5k78\") pod \"insights-runtime-extractor-nk2zz\" (UID: \"7ceb0bb1-def2-44a5-9e31-ceffd848125d\") " pod="openshift-insights/insights-runtime-extractor-nk2zz"
Oct 30 13:03:02.138253 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.138235    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsj7z\" (UniqueName: \"kubernetes.io/projected/155bbdd4-b566-467f-88c6-ec5b4b881a09-kube-api-access-lsj7z\") pod \"insights-operator-7c4986d449-lgssx\" (UID: \"155bbdd4-b566-467f-88c6-ec5b4b881a09\") " pod="openshift-insights/insights-operator-7c4986d449-lgssx"
Oct 30 13:03:02.139721 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.139699    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-789vz\" (UniqueName: \"kubernetes.io/projected/7d29458a-da6c-461c-9941-afa323061f30-kube-api-access-789vz\") pod \"service-ca-5fffc97bdf-x8ck2\" (UID: \"7d29458a-da6c-461c-9941-afa323061f30\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x8ck2"
Oct 30 13:03:02.139788 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.139753    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjjr5\" (UniqueName: \"kubernetes.io/projected/44f513ab-9115-4761-aa8b-c546212ec17b-kube-api-access-vjjr5\") pod \"kas-connection-checker-685895c7c9-wnzh2\" (UID: \"44f513ab-9115-4761-aa8b-c546212ec17b\") " pod="kube-system/kas-connection-checker-685895c7c9-wnzh2"
Oct 30 13:03:02.139824 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.139809    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xws2p\" (UniqueName: \"kubernetes.io/projected/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-kube-api-access-xws2p\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.139917 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.139901    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncqh7\" (UniqueName: \"kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-kube-api-access-ncqh7\") pod \"image-registry-5b97bd7c47-dssmx\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") " pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:03:02.140415 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.140399    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jdqx\" (UniqueName: \"kubernetes.io/projected/7bbbc7e0-1790-459b-952a-b6b7b4d55113-kube-api-access-8jdqx\") pod \"migrator-66df5d496b-r5pcv\" (UID: \"7bbbc7e0-1790-459b-952a-b6b7b4d55113\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-r5pcv"
Oct 30 13:03:02.140472 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.140405    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdrpm\" (UniqueName: \"kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-kube-api-access-vdrpm\") pod \"image-registry-9677cf66b-kpdfl\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") " pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:03:02.140642 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.140624    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsfql\" (UniqueName: \"kubernetes.io/projected/a32a253a-ecaf-4f90-af1c-27a7e53b9629-kube-api-access-qsfql\") pod \"network-metrics-daemon-mbncl\" (UID: \"a32a253a-ecaf-4f90-af1c-27a7e53b9629\") " pod="openshift-multus/network-metrics-daemon-mbncl"
Oct 30 13:03:02.140965 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.140950    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4kjx\" (UniqueName: \"kubernetes.io/projected/3c7ab011-8fd4-4d6d-b3f2-72d6af11d3e1-kube-api-access-p4kjx\") pod \"migrator-66df5d496b-k6rlj\" (UID: \"3c7ab011-8fd4-4d6d-b3f2-72d6af11d3e1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-k6rlj"
Oct 30 13:03:02.141423 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.141407    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgqhj\" (UniqueName: \"kubernetes.io/projected/0c8e72ed-d54d-4e6a-bf66-65a16430351d-kube-api-access-dgqhj\") pod \"iptables-alerter-5swg6\" (UID: \"0c8e72ed-d54d-4e6a-bf66-65a16430351d\") " pod="openshift-network-operator/iptables-alerter-5swg6"
Oct 30 13:03:02.141687 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.141670    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-bound-sa-token\") pod \"image-registry-9677cf66b-kpdfl\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") " pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:03:02.141720 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.141672    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbpg6\" (UniqueName: \"kubernetes.io/projected/46bedd0c-7237-4128-be98-69e4474bb535-kube-api-access-kbpg6\") pod \"kas-connection-checker-685895c7c9-22fm4\" (UID: \"46bedd0c-7237-4128-be98-69e4474bb535\") " pod="kube-system/kas-connection-checker-685895c7c9-22fm4"
Oct 30 13:03:02.142085 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.142069    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rflhr\" (UniqueName: \"kubernetes.io/projected/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-kube-api-access-rflhr\") pod \"router-default-668b54766c-wgk28\" (UID: \"d5db3ba8-f1de-4fa5-9839-e7e8200da58f\") " pod="openshift-ingress/router-default-668b54766c-wgk28"
Oct 30 13:03:02.218355 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.218263    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-node-log\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.218355 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.218301    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-host-cni-bin\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.218355 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.218324    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-etc-sysctl-conf\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:03:02.218355 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.218349    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-var-lib-openvswitch\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.218646 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.218381    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/183f23e7-ae80-445b-960c-fff9b410ba5b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lrqnm\" (UID: \"183f23e7-ae80-445b-960c-fff9b410ba5b\") " pod="openshift-multus/multus-additional-cni-plugins-lrqnm"
Oct 30 13:03:02.218646 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.218380    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-host-cni-bin\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.218646 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.218404    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/10b774a1-1476-4f91-b6e5-d52b3f45700e-kubelet-config\") pod \"global-pull-secret-syncer-wrfqz\" (UID: \"10b774a1-1476-4f91-b6e5-d52b3f45700e\") " pod="kube-system/global-pull-secret-syncer-wrfqz"
Oct 30 13:03:02.218646 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.218424    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-sys\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:03:02.218646 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.218380    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-node-log\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.218646 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.218448    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-host-var-lib-cni-bin\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:03:02.218646 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.218449    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/183f23e7-ae80-445b-960c-fff9b410ba5b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lrqnm\" (UID: \"183f23e7-ae80-445b-960c-fff9b410ba5b\") " pod="openshift-multus/multus-additional-cni-plugins-lrqnm"
Oct 30 13:03:02.218646 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.218496    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-sys\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:03:02.218646 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.218506    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/10b774a1-1476-4f91-b6e5-d52b3f45700e-kubelet-config\") pod \"global-pull-secret-syncer-wrfqz\" (UID: \"10b774a1-1476-4f91-b6e5-d52b3f45700e\") " pod="kube-system/global-pull-secret-syncer-wrfqz"
Oct 30 13:03:02.218646 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.218507    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-var-lib-openvswitch\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.218646 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.218528    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a323e312-dd91-486f-985d-92b9ab8cca04-device-dir\") pod \"aws-ebs-csi-driver-node-9q945\" (UID: \"a323e312-dd91-486f-985d-92b9ab8cca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q945"
Oct 30 13:03:02.218646 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.218541    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-host-var-lib-cni-bin\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:03:02.218646 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.218554    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0c8e72ed-d54d-4e6a-bf66-65a16430351d-host-slash\") pod \"iptables-alerter-5swg6\" (UID: \"0c8e72ed-d54d-4e6a-bf66-65a16430351d\") " pod="openshift-network-operator/iptables-alerter-5swg6"
Oct 30 13:03:02.218646 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.218572    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e640effb-1ff4-40f2-8d7b-53732edcd2bf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-x8wqt\" (UID: \"e640effb-1ff4-40f2-8d7b-53732edcd2bf\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x8wqt"
Oct 30 13:03:02.218646 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.218517    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-etc-sysctl-conf\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:03:02.218646 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.218575    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a323e312-dd91-486f-985d-92b9ab8cca04-device-dir\") pod \"aws-ebs-csi-driver-node-9q945\" (UID: \"a323e312-dd91-486f-985d-92b9ab8cca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q945"
Oct 30 13:03:02.218646 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.218595    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7ceb0bb1-def2-44a5-9e31-ceffd848125d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nk2zz\" (UID: \"7ceb0bb1-def2-44a5-9e31-ceffd848125d\") " pod="openshift-insights/insights-runtime-extractor-nk2zz"
Oct 30 13:03:02.218646 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.218621    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0c8e72ed-d54d-4e6a-bf66-65a16430351d-host-slash\") pod \"iptables-alerter-5swg6\" (UID: \"0c8e72ed-d54d-4e6a-bf66-65a16430351d\") " pod="openshift-network-operator/iptables-alerter-5swg6"
Oct 30 13:03:02.218646 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.218633    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a32a253a-ecaf-4f90-af1c-27a7e53b9629-metrics-certs\") pod \"network-metrics-daemon-mbncl\" (UID: \"a32a253a-ecaf-4f90-af1c-27a7e53b9629\") " pod="openshift-multus/network-metrics-daemon-mbncl"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:02.218672    5093 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.218670    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-systemd-units\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.218694    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-systemd-units\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.218708    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/087058f5-e286-4ac7-8ed8-a280da332878-host\") pod \"node-ca-f7mvz\" (UID: \"087058f5-e286-4ac7-8ed8-a280da332878\") " pod="openshift-image-registry/node-ca-f7mvz"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.218732    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-log-socket\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:02.218673    5093 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:02.218753    5093 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ceb0bb1-def2-44a5-9e31-ceffd848125d-insights-runtime-extractor-tls podName:7ceb0bb1-def2-44a5-9e31-ceffd848125d nodeName:}" failed. No retries permitted until 2026-10-30 13:03:02.718736943 +0000 UTC m=+1.866004192 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7ceb0bb1-def2-44a5-9e31-ceffd848125d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nk2zz" (UID: "7ceb0bb1-def2-44a5-9e31-ceffd848125d") : secret "insights-runtime-extractor-tls" not found
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.218767    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-log-socket\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.218777    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-multus-cni-dir\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:02.218796    5093 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e640effb-1ff4-40f2-8d7b-53732edcd2bf-cluster-monitoring-operator-tls podName:e640effb-1ff4-40f2-8d7b-53732edcd2bf nodeName:}" failed. No retries permitted until 2026-10-30 13:03:02.718776993 +0000 UTC m=+1.866044216 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e640effb-1ff4-40f2-8d7b-53732edcd2bf-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-x8wqt" (UID: "e640effb-1ff4-40f2-8d7b-53732edcd2bf") : secret "cluster-monitoring-operator-tls" not found
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.218804    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/087058f5-e286-4ac7-8ed8-a280da332878-host\") pod \"node-ca-f7mvz\" (UID: \"087058f5-e286-4ac7-8ed8-a280da332878\") " pod="openshift-image-registry/node-ca-f7mvz"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.218822    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-lib-modules\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.218834    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-multus-cni-dir\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.218840    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-host-run-multus-certs\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.218864    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-host-run-multus-certs\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.218874    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-multus-socket-dir-parent\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.218901    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-host-cni-netd\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.218919    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-multus-socket-dir-parent\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.218924    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/10b774a1-1476-4f91-b6e5-d52b3f45700e-dbus\") pod \"global-pull-secret-syncer-wrfqz\" (UID: \"10b774a1-1476-4f91-b6e5-d52b3f45700e\") " pod="kube-system/global-pull-secret-syncer-wrfqz"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.218953    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a323e312-dd91-486f-985d-92b9ab8cca04-sys-fs\") pod \"aws-ebs-csi-driver-node-9q945\" (UID: \"a323e312-dd91-486f-985d-92b9ab8cca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q945"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.218957    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-host-cni-netd\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.218954    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-lib-modules\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.218979    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-etc-sysconfig\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.218991    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/10b774a1-1476-4f91-b6e5-d52b3f45700e-dbus\") pod \"global-pull-secret-syncer-wrfqz\" (UID: \"10b774a1-1476-4f91-b6e5-d52b3f45700e\") " pod="kube-system/global-pull-secret-syncer-wrfqz"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219004    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a323e312-dd91-486f-985d-92b9ab8cca04-sys-fs\") pod \"aws-ebs-csi-driver-node-9q945\" (UID: \"a323e312-dd91-486f-985d-92b9ab8cca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q945"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219012    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-var-lib-kubelet\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219018    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-etc-sysconfig\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219041    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a323e312-dd91-486f-985d-92b9ab8cca04-registration-dir\") pod \"aws-ebs-csi-driver-node-9q945\" (UID: \"a323e312-dd91-486f-985d-92b9ab8cca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q945"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219068    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-host\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219094    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a323e312-dd91-486f-985d-92b9ab8cca04-registration-dir\") pod \"aws-ebs-csi-driver-node-9q945\" (UID: \"a323e312-dd91-486f-985d-92b9ab8cca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q945"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219098    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-var-lib-kubelet\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219093    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-etc-kubernetes\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219143    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-host\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219145    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-etc-kubernetes\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219151    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-service-ca-bundle\") pod \"router-default-668b54766c-wgk28\" (UID: \"d5db3ba8-f1de-4fa5-9839-e7e8200da58f\") " pod="openshift-ingress/router-default-668b54766c-wgk28"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:02.219164    5093 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219176    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-host-run-k8s-cni-cncf-io\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219195    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-host-slash\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:02.219216    5093 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a32a253a-ecaf-4f90-af1c-27a7e53b9629-metrics-certs podName:a32a253a-ecaf-4f90-af1c-27a7e53b9629 nodeName:}" failed. No retries permitted until 2026-10-30 13:03:02.719200732 +0000 UTC m=+1.866467965 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a32a253a-ecaf-4f90-af1c-27a7e53b9629-metrics-certs") pod "network-metrics-daemon-mbncl" (UID: "a32a253a-ecaf-4f90-af1c-27a7e53b9629") : secret "metrics-daemon-secret" not found
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219216    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-host-slash\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219234    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-host-run-k8s-cni-cncf-io\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:02.219246    5093 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-service-ca-bundle podName:d5db3ba8-f1de-4fa5-9839-e7e8200da58f nodeName:}" failed. No retries permitted until 2026-10-30 13:03:02.719235777 +0000 UTC m=+1.866503012 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-service-ca-bundle") pod "router-default-668b54766c-wgk28" (UID: "d5db3ba8-f1de-4fa5-9839-e7e8200da58f") : configmap references non-existent config key: service-ca.crt
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219267    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-run-openvswitch\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219296    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/183f23e7-ae80-445b-960c-fff9b410ba5b-cnibin\") pod \"multus-additional-cni-plugins-lrqnm\" (UID: \"183f23e7-ae80-445b-960c-fff9b410ba5b\") " pod="openshift-multus/multus-additional-cni-plugins-lrqnm"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219320    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/183f23e7-ae80-445b-960c-fff9b410ba5b-cnibin\") pod \"multus-additional-cni-plugins-lrqnm\" (UID: \"183f23e7-ae80-445b-960c-fff9b410ba5b\") " pod="openshift-multus/multus-additional-cni-plugins-lrqnm"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219324    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a323e312-dd91-486f-985d-92b9ab8cca04-etc-selinux\") pod \"aws-ebs-csi-driver-node-9q945\" (UID: \"a323e312-dd91-486f-985d-92b9ab8cca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q945"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219298    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-run-openvswitch\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219365    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-host-run-netns\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219375    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a323e312-dd91-486f-985d-92b9ab8cca04-etc-selinux\") pod \"aws-ebs-csi-driver-node-9q945\" (UID: \"a323e312-dd91-486f-985d-92b9ab8cca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q945"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219396    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-etc-systemd\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219411    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-host-run-netns\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219421    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-host-kubelet\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219447    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a323e312-dd91-486f-985d-92b9ab8cca04-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9q945\" (UID: \"a323e312-dd91-486f-985d-92b9ab8cca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q945"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219453    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-etc-systemd\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219471    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d0b2655-d52d-4c95-9589-d45237fa1057-metrics-tls\") pod \"dns-default-8nb7s\" (UID: \"0d0b2655-d52d-4c95-9589-d45237fa1057\") " pod="openshift-dns/dns-default-8nb7s"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219509    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-host-kubelet\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:02.219527    5093 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219538    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-host-run-ovn-kubernetes\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219545    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a323e312-dd91-486f-985d-92b9ab8cca04-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9q945\" (UID: \"a323e312-dd91-486f-985d-92b9ab8cca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q945"
Oct 30 13:03:02.219516 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:02.219562    5093 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d0b2655-d52d-4c95-9589-d45237fa1057-metrics-tls podName:0d0b2655-d52d-4c95-9589-d45237fa1057 nodeName:}" failed. No retries permitted until 2026-10-30 13:03:02.719551216 +0000 UTC m=+1.866818454 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0d0b2655-d52d-4c95-9589-d45237fa1057-metrics-tls") pod "dns-default-8nb7s" (UID: "0d0b2655-d52d-4c95-9589-d45237fa1057") : secret "dns-default-metrics-tls" not found
Oct 30 13:03:02.222075 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219574    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-etc-modprobe-d\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:03:02.222075 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219580    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-host-run-ovn-kubernetes\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.222075 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219590    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-run\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:03:02.222075 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219607    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7cca8a39-bee0-4705-bcde-b31060e44944-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gxcrm\" (UID: \"7cca8a39-bee0-4705-bcde-b31060e44944\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gxcrm"
Oct 30 13:03:02.222075 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219624    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a323e312-dd91-486f-985d-92b9ab8cca04-socket-dir\") pod \"aws-ebs-csi-driver-node-9q945\" (UID: \"a323e312-dd91-486f-985d-92b9ab8cca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q945"
Oct 30 13:03:02.222075 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219639    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.222075 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219654    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-host-var-lib-cni-multus\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:03:02.222075 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219659    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-etc-modprobe-d\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:03:02.222075 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219697    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-host-var-lib-cni-multus\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:03:02.222075 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219702    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a323e312-dd91-486f-985d-92b9ab8cca04-socket-dir\") pod \"aws-ebs-csi-driver-node-9q945\" (UID: \"a323e312-dd91-486f-985d-92b9ab8cca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q945"
Oct 30 13:03:02.222075 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219705    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.222075 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219710    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-run\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:03:02.222075 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219728    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-registry-tls\") pod \"image-registry-5b97bd7c47-dssmx\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") " pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:03:02.222075 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219750    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-registry-tls\") pod \"image-registry-9677cf66b-kpdfl\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") " pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:03:02.222075 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219777    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5a16e7b2-e45d-480a-a7bc-d914640b9867-hosts-file\") pod \"node-resolver-gltng\" (UID: \"5a16e7b2-e45d-480a-a7bc-d914640b9867\") " pod="openshift-dns/node-resolver-gltng"
Oct 30 13:03:02.222075 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219803    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-metrics-certs\") pod \"router-default-668b54766c-wgk28\" (UID: \"d5db3ba8-f1de-4fa5-9839-e7e8200da58f\") " pod="openshift-ingress/router-default-668b54766c-wgk28"
Oct 30 13:03:02.222075 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:02.219819    5093 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 13:03:02.222075 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:02.219829    5093 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9677cf66b-kpdfl: secret "image-registry-tls" not found
Oct 30 13:03:02.222075 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219830    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-host-var-lib-kubelet\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:03:02.222075 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219858    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-hostroot\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:03:02.222075 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219879    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5a16e7b2-e45d-480a-a7bc-d914640b9867-hosts-file\") pod \"node-resolver-gltng\" (UID: \"5a16e7b2-e45d-480a-a7bc-d914640b9867\") " pod="openshift-dns/node-resolver-gltng"
Oct 30 13:03:02.222075 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219886    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-host-var-lib-kubelet\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:03:02.222075 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219897    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-hostroot\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:03:02.222075 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:02.219833    5093 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 13:03:02.222075 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:02.219915    5093 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b97bd7c47-dssmx: secret "image-registry-tls" not found
Oct 30 13:03:02.222075 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:02.219885    5093 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-registry-tls podName:b4fd1eb6-625a-4649-ba27-9e8798fd1bc5 nodeName:}" failed. No retries permitted until 2026-10-30 13:03:02.71987143 +0000 UTC m=+1.867138674 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-registry-tls") pod "image-registry-9677cf66b-kpdfl" (UID: "b4fd1eb6-625a-4649-ba27-9e8798fd1bc5") : secret "image-registry-tls" not found
Oct 30 13:03:02.222075 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219958    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-multus-conf-dir\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:03:02.222075 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219982    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-etc-openvswitch\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.222075 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.219983    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-multus-conf-dir\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:03:02.222075 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:02.220004    5093 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-registry-tls podName:feda4324-4a5f-491b-a04c-202c222261a0 nodeName:}" failed. No retries permitted until 2026-10-30 13:03:02.719988141 +0000 UTC m=+1.867255367 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-registry-tls") pod "image-registry-5b97bd7c47-dssmx" (UID: "feda4324-4a5f-491b-a04c-202c222261a0") : secret "image-registry-tls" not found
Oct 30 13:03:02.222075 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.220008    5093 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 13:03:02.222075 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.220003    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-etc-openvswitch\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.222075 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.220072    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-etc-kubernetes\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:03:02.222075 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.220098    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-cnibin\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:03:02.222075 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.220124    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-run-systemd\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.222075 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.220161    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-cnibin\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:03:02.222075 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.220175    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-run-systemd\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.222075 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.220181    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7ceb0bb1-def2-44a5-9e31-ceffd848125d-crio-socket\") pod \"insights-runtime-extractor-nk2zz\" (UID: \"7ceb0bb1-def2-44a5-9e31-ceffd848125d\") " pod="openshift-insights/insights-runtime-extractor-nk2zz"
Oct 30 13:03:02.222075 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.220212    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-run-ovn\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.222075 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.220145    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-etc-kubernetes\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:03:02.222075 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.220280    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-system-cni-dir\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:03:02.222075 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.220870    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3a675ac7-fbe1-449b-accf-bb0456ed0bd3-run-ovn\") pod \"ovnkube-node-67tnm\" (UID: \"3a675ac7-fbe1-449b-accf-bb0456ed0bd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:02.222075 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.221067    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7ceb0bb1-def2-44a5-9e31-ceffd848125d-crio-socket\") pod \"insights-runtime-extractor-nk2zz\" (UID: \"7ceb0bb1-def2-44a5-9e31-ceffd848125d\") " pod="openshift-insights/insights-runtime-extractor-nk2zz"
Oct 30 13:03:02.222075 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.221107    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-system-cni-dir\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:03:02.223962 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.222443    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-host-run-netns\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:03:02.223962 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.222488    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/63086243-e82d-4fc5-bfef-e4241c22ad4e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2w8jt\" (UID: \"63086243-e82d-4fc5-bfef-e4241c22ad4e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2w8jt"
Oct 30 13:03:02.223962 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.222525    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/183f23e7-ae80-445b-960c-fff9b410ba5b-system-cni-dir\") pod \"multus-additional-cni-plugins-lrqnm\" (UID: \"183f23e7-ae80-445b-960c-fff9b410ba5b\") " pod="openshift-multus/multus-additional-cni-plugins-lrqnm"
Oct 30 13:03:02.223962 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.222559    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-etc-sysctl-d\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:03:02.223962 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.222734    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f1fb4abc-b161-461c-b8ef-bd1212010510-etc-sysctl-d\") pod \"tuned-s9vk9\" (UID: \"f1fb4abc-b161-461c-b8ef-bd1212010510\") " pod="openshift-cluster-node-tuning-operator/tuned-s9vk9"
Oct 30 13:03:02.223962 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.222784    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de69ec51-fd3a-4bc4-ac17-ff02bc582905-host-run-netns\") pod \"multus-f2bqp\" (UID: \"de69ec51-fd3a-4bc4-ac17-ff02bc582905\") " pod="openshift-multus/multus-f2bqp"
Oct 30 13:03:02.223962 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.222840    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-metrics-certs\") pod \"router-default-668b54766c-wgk28\" (UID: \"d5db3ba8-f1de-4fa5-9839-e7e8200da58f\") " pod="openshift-ingress/router-default-668b54766c-wgk28"
Oct 30 13:03:02.223962 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.222932    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/183f23e7-ae80-445b-960c-fff9b410ba5b-system-cni-dir\") pod \"multus-additional-cni-plugins-lrqnm\" (UID: \"183f23e7-ae80-445b-960c-fff9b410ba5b\") " pod="openshift-multus/multus-additional-cni-plugins-lrqnm"
Oct 30 13:03:02.226819 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.226670    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7cca8a39-bee0-4705-bcde-b31060e44944-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gxcrm\" (UID: \"7cca8a39-bee0-4705-bcde-b31060e44944\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gxcrm"
Oct 30 13:03:02.228263 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.228242    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/63086243-e82d-4fc5-bfef-e4241c22ad4e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2w8jt\" (UID: \"63086243-e82d-4fc5-bfef-e4241c22ad4e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2w8jt"
Oct 30 13:03:02.299492 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.299464    5093 scope.go:122] "RemoveContainer" containerID="843f19531f8ed50dca559ae32fd0ed454c48aa876303554a6b13044481fee1a7"
Oct 30 13:03:02.300841 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.300816    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gxcrm"
Oct 30 13:03:02.302642 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.302618    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-2w8jt"
Oct 30 13:03:02.610786 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.610715    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gxcrm"]
Oct 30 13:03:02.620457 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.620429    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-2w8jt"]
Oct 30 13:03:02.641423 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:02.641381    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63086243_e82d_4fc5_bfef_e4241c22ad4e.slice/crio-d9012f136e1b69387d8d36b1fda285a96cfdfba03f60c3cd0fc02fc4fb3bd794 WatchSource:0}: Error finding container d9012f136e1b69387d8d36b1fda285a96cfdfba03f60c3cd0fc02fc4fb3bd794: Status 404 returned error can't find the container with id d9012f136e1b69387d8d36b1fda285a96cfdfba03f60c3cd0fc02fc4fb3bd794
Oct 30 13:03:02.643747 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.643725    5093 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 13:03:02.727103 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.727073    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-service-ca-bundle\") pod \"router-default-668b54766c-wgk28\" (UID: \"d5db3ba8-f1de-4fa5-9839-e7e8200da58f\") " pod="openshift-ingress/router-default-668b54766c-wgk28"
Oct 30 13:03:02.727263 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.727115    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d0b2655-d52d-4c95-9589-d45237fa1057-metrics-tls\") pod \"dns-default-8nb7s\" (UID: \"0d0b2655-d52d-4c95-9589-d45237fa1057\") " pod="openshift-dns/dns-default-8nb7s"
Oct 30 13:03:02.727263 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.727170    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-registry-tls\") pod \"image-registry-5b97bd7c47-dssmx\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") " pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:03:02.727263 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.727188    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-registry-tls\") pod \"image-registry-9677cf66b-kpdfl\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") " pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:03:02.727435 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:02.727258    5093 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-service-ca-bundle podName:d5db3ba8-f1de-4fa5-9839-e7e8200da58f nodeName:}" failed. No retries permitted until 2026-10-30 13:03:03.727236118 +0000 UTC m=+2.874503341 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-service-ca-bundle") pod "router-default-668b54766c-wgk28" (UID: "d5db3ba8-f1de-4fa5-9839-e7e8200da58f") : configmap references non-existent config key: service-ca.crt
Oct 30 13:03:02.727435 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.727292    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e640effb-1ff4-40f2-8d7b-53732edcd2bf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-x8wqt\" (UID: \"e640effb-1ff4-40f2-8d7b-53732edcd2bf\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x8wqt"
Oct 30 13:03:02.727435 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:02.727301    5093 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 13:03:02.727435 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:02.727352    5093 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9677cf66b-kpdfl: secret "image-registry-tls" not found
Oct 30 13:03:02.727435 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.727332    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7ceb0bb1-def2-44a5-9e31-ceffd848125d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nk2zz\" (UID: \"7ceb0bb1-def2-44a5-9e31-ceffd848125d\") " pod="openshift-insights/insights-runtime-extractor-nk2zz"
Oct 30 13:03:02.727435 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:02.727383    5093 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 13:03:02.727435 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:02.727412    5093 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-registry-tls podName:b4fd1eb6-625a-4649-ba27-9e8798fd1bc5 nodeName:}" failed. No retries permitted until 2026-10-30 13:03:03.72739992 +0000 UTC m=+2.874667143 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-registry-tls") pod "image-registry-9677cf66b-kpdfl" (UID: "b4fd1eb6-625a-4649-ba27-9e8798fd1bc5") : secret "image-registry-tls" not found
Oct 30 13:03:02.727435 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:02.727412    5093 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 13:03:02.727435 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:02.727430    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a32a253a-ecaf-4f90-af1c-27a7e53b9629-metrics-certs\") pod \"network-metrics-daemon-mbncl\" (UID: \"a32a253a-ecaf-4f90-af1c-27a7e53b9629\") " pod="openshift-multus/network-metrics-daemon-mbncl"
Oct 30 13:03:02.727435 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:02.727437    5093 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e640effb-1ff4-40f2-8d7b-53732edcd2bf-cluster-monitoring-operator-tls podName:e640effb-1ff4-40f2-8d7b-53732edcd2bf nodeName:}" failed. No retries permitted until 2026-10-30 13:03:03.727426772 +0000 UTC m=+2.874694020 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e640effb-1ff4-40f2-8d7b-53732edcd2bf-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-x8wqt" (UID: "e640effb-1ff4-40f2-8d7b-53732edcd2bf") : secret "cluster-monitoring-operator-tls" not found
Oct 30 13:03:02.727435 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:02.727312    5093 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 13:03:02.727770 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:02.727308    5093 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 13:03:02.727770 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:02.727456    5093 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ceb0bb1-def2-44a5-9e31-ceffd848125d-insights-runtime-extractor-tls podName:7ceb0bb1-def2-44a5-9e31-ceffd848125d nodeName:}" failed. No retries permitted until 2026-10-30 13:03:03.72744978 +0000 UTC m=+2.874717000 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7ceb0bb1-def2-44a5-9e31-ceffd848125d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nk2zz" (UID: "7ceb0bb1-def2-44a5-9e31-ceffd848125d") : secret "insights-runtime-extractor-tls" not found
Oct 30 13:03:02.727770 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:02.727455    5093 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b97bd7c47-dssmx: secret "image-registry-tls" not found
Oct 30 13:03:02.727770 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:02.727495    5093 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 13:03:02.727770 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:02.727532    5093 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d0b2655-d52d-4c95-9589-d45237fa1057-metrics-tls podName:0d0b2655-d52d-4c95-9589-d45237fa1057 nodeName:}" failed. No retries permitted until 2026-10-30 13:03:03.72752124 +0000 UTC m=+2.874788470 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0d0b2655-d52d-4c95-9589-d45237fa1057-metrics-tls") pod "dns-default-8nb7s" (UID: "0d0b2655-d52d-4c95-9589-d45237fa1057") : secret "dns-default-metrics-tls" not found
Oct 30 13:03:02.727770 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:02.727546    5093 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-registry-tls podName:feda4324-4a5f-491b-a04c-202c222261a0 nodeName:}" failed. No retries permitted until 2026-10-30 13:03:03.727540438 +0000 UTC m=+2.874807657 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-registry-tls") pod "image-registry-5b97bd7c47-dssmx" (UID: "feda4324-4a5f-491b-a04c-202c222261a0") : secret "image-registry-tls" not found
Oct 30 13:03:02.727770 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:02.727562    5093 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a32a253a-ecaf-4f90-af1c-27a7e53b9629-metrics-certs podName:a32a253a-ecaf-4f90-af1c-27a7e53b9629 nodeName:}" failed. No retries permitted until 2026-10-30 13:03:03.727557046 +0000 UTC m=+2.874824264 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a32a253a-ecaf-4f90-af1c-27a7e53b9629-metrics-certs") pod "network-metrics-daemon-mbncl" (UID: "a32a253a-ecaf-4f90-af1c-27a7e53b9629") : secret "metrics-daemon-secret" not found
Oct 30 13:03:03.071287 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:03.071257    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jdj4_f93ee017-ab79-447c-90e0-5b85b71dba24/console-operator/1.log"
Oct 30 13:03:03.071754 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:03.071394    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-6jdj4" event={"ID":"f93ee017-ab79-447c-90e0-5b85b71dba24","Type":"ContainerStarted","Data":"de89c5abe098668bbfdb6d264435076ee7b2d6993e0326a4840a03c89f685343"}
Oct 30 13:03:03.072904 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:03.072871    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-2w8jt" event={"ID":"63086243-e82d-4fc5-bfef-e4241c22ad4e","Type":"ContainerStarted","Data":"d9012f136e1b69387d8d36b1fda285a96cfdfba03f60c3cd0fc02fc4fb3bd794"}
Oct 30 13:03:03.075949 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:03.075918    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gxcrm" event={"ID":"7cca8a39-bee0-4705-bcde-b31060e44944","Type":"ContainerStarted","Data":"1894e39ca4d6f930eade736ccbb2eec1c4d1b28852f56f94c53251738ea26477"}
Oct 30 13:03:03.737508 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:03.737474    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e640effb-1ff4-40f2-8d7b-53732edcd2bf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-x8wqt\" (UID: \"e640effb-1ff4-40f2-8d7b-53732edcd2bf\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x8wqt"
Oct 30 13:03:03.737610 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:03.737523    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7ceb0bb1-def2-44a5-9e31-ceffd848125d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nk2zz\" (UID: \"7ceb0bb1-def2-44a5-9e31-ceffd848125d\") " pod="openshift-insights/insights-runtime-extractor-nk2zz"
Oct 30 13:03:03.737610 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:03.737547    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a32a253a-ecaf-4f90-af1c-27a7e53b9629-metrics-certs\") pod \"network-metrics-daemon-mbncl\" (UID: \"a32a253a-ecaf-4f90-af1c-27a7e53b9629\") " pod="openshift-multus/network-metrics-daemon-mbncl"
Oct 30 13:03:03.737700 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:03.737631    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-service-ca-bundle\") pod \"router-default-668b54766c-wgk28\" (UID: \"d5db3ba8-f1de-4fa5-9839-e7e8200da58f\") " pod="openshift-ingress/router-default-668b54766c-wgk28"
Oct 30 13:03:03.737700 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:03.737676    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d0b2655-d52d-4c95-9589-d45237fa1057-metrics-tls\") pod \"dns-default-8nb7s\" (UID: \"0d0b2655-d52d-4c95-9589-d45237fa1057\") " pod="openshift-dns/dns-default-8nb7s"
Oct 30 13:03:03.737792 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:03.737710    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-registry-tls\") pod \"image-registry-5b97bd7c47-dssmx\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") " pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:03:03.737792 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:03.737731    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-registry-tls\") pod \"image-registry-9677cf66b-kpdfl\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") " pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:03:03.737886 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:03.737858    5093 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 13:03:03.737886 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:03.737869    5093 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9677cf66b-kpdfl: secret "image-registry-tls" not found
Oct 30 13:03:03.737982 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:03.737920    5093 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-registry-tls podName:b4fd1eb6-625a-4649-ba27-9e8798fd1bc5 nodeName:}" failed. No retries permitted until 2026-10-30 13:03:05.737903014 +0000 UTC m=+4.885170245 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-registry-tls") pod "image-registry-9677cf66b-kpdfl" (UID: "b4fd1eb6-625a-4649-ba27-9e8798fd1bc5") : secret "image-registry-tls" not found
Oct 30 13:03:03.738830 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:03.738620    5093 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-service-ca-bundle podName:d5db3ba8-f1de-4fa5-9839-e7e8200da58f nodeName:}" failed. No retries permitted until 2026-10-30 13:03:05.738599132 +0000 UTC m=+4.885866355 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-service-ca-bundle") pod "router-default-668b54766c-wgk28" (UID: "d5db3ba8-f1de-4fa5-9839-e7e8200da58f") : configmap references non-existent config key: service-ca.crt
Oct 30 13:03:03.738830 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:03.738653    5093 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 13:03:03.738830 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:03.738667    5093 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b97bd7c47-dssmx: secret "image-registry-tls" not found
Oct 30 13:03:03.738830 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:03.738710    5093 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-registry-tls podName:feda4324-4a5f-491b-a04c-202c222261a0 nodeName:}" failed. No retries permitted until 2026-10-30 13:03:05.738694013 +0000 UTC m=+4.885961252 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-registry-tls") pod "image-registry-5b97bd7c47-dssmx" (UID: "feda4324-4a5f-491b-a04c-202c222261a0") : secret "image-registry-tls" not found
Oct 30 13:03:03.738830 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:03.738709    5093 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 13:03:03.738830 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:03.738741    5093 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ceb0bb1-def2-44a5-9e31-ceffd848125d-insights-runtime-extractor-tls podName:7ceb0bb1-def2-44a5-9e31-ceffd848125d nodeName:}" failed. No retries permitted until 2026-10-30 13:03:05.738735597 +0000 UTC m=+4.886002863 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7ceb0bb1-def2-44a5-9e31-ceffd848125d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nk2zz" (UID: "7ceb0bb1-def2-44a5-9e31-ceffd848125d") : secret "insights-runtime-extractor-tls" not found
Oct 30 13:03:03.738830 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:03.738758    5093 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 13:03:03.738830 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:03.738789    5093 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 13:03:03.738830 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:03.738796    5093 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a32a253a-ecaf-4f90-af1c-27a7e53b9629-metrics-certs podName:a32a253a-ecaf-4f90-af1c-27a7e53b9629 nodeName:}" failed. No retries permitted until 2026-10-30 13:03:05.738783216 +0000 UTC m=+4.886050454 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a32a253a-ecaf-4f90-af1c-27a7e53b9629-metrics-certs") pod "network-metrics-daemon-mbncl" (UID: "a32a253a-ecaf-4f90-af1c-27a7e53b9629") : secret "metrics-daemon-secret" not found
Oct 30 13:03:03.738830 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:03.738814    5093 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d0b2655-d52d-4c95-9589-d45237fa1057-metrics-tls podName:0d0b2655-d52d-4c95-9589-d45237fa1057 nodeName:}" failed. No retries permitted until 2026-10-30 13:03:05.738806593 +0000 UTC m=+4.886073853 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0d0b2655-d52d-4c95-9589-d45237fa1057-metrics-tls") pod "dns-default-8nb7s" (UID: "0d0b2655-d52d-4c95-9589-d45237fa1057") : secret "dns-default-metrics-tls" not found
Oct 30 13:03:03.741108 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:03.741081    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e640effb-1ff4-40f2-8d7b-53732edcd2bf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-x8wqt\" (UID: \"e640effb-1ff4-40f2-8d7b-53732edcd2bf\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x8wqt"
Oct 30 13:03:03.800267 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:03.800235    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x8wqt"
Oct 30 13:03:04.000903 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:04.000827    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:04.082342 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:04.082305    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-2w8jt" event={"ID":"63086243-e82d-4fc5-bfef-e4241c22ad4e","Type":"ContainerStarted","Data":"b21ecb04716830ed00fef9b280eb3de23b4c90dcd1f7f24d857c1c9b44099a35"}
Oct 30 13:03:04.096201 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:04.096166    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-x8wqt"]
Oct 30 13:03:04.100217 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:04.100191    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode640effb_1ff4_40f2_8d7b_53732edcd2bf.slice/crio-fb4ec164eeeac1f878adf593af363b9eea1f4c5a92474ffbeb0d23da5b2a4729 WatchSource:0}: Error finding container fb4ec164eeeac1f878adf593af363b9eea1f4c5a92474ffbeb0d23da5b2a4729: Status 404 returned error can't find the container with id fb4ec164eeeac1f878adf593af363b9eea1f4c5a92474ffbeb0d23da5b2a4729
Oct 30 13:03:05.088603 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:05.088568    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gxcrm" event={"ID":"7cca8a39-bee0-4705-bcde-b31060e44944","Type":"ContainerStarted","Data":"b67a76018e748e1b8c2e300e8aaa1fe5056ea45d21bd28c4569075bae4c29585"}
Oct 30 13:03:05.089087 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:05.088628    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gxcrm" event={"ID":"7cca8a39-bee0-4705-bcde-b31060e44944","Type":"ContainerStarted","Data":"e2613e80cfd468c638313e249a89a2355ba19cbebf7dba1ffb9894c2b62e6adb"}
Oct 30 13:03:05.089983 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:05.089952    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x8wqt" event={"ID":"e640effb-1ff4-40f2-8d7b-53732edcd2bf","Type":"ContainerStarted","Data":"fb4ec164eeeac1f878adf593af363b9eea1f4c5a92474ffbeb0d23da5b2a4729"}
Oct 30 13:03:05.758401 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:05.758360    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-service-ca-bundle\") pod \"router-default-668b54766c-wgk28\" (UID: \"d5db3ba8-f1de-4fa5-9839-e7e8200da58f\") " pod="openshift-ingress/router-default-668b54766c-wgk28"
Oct 30 13:03:05.758579 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:05.758420    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d0b2655-d52d-4c95-9589-d45237fa1057-metrics-tls\") pod \"dns-default-8nb7s\" (UID: \"0d0b2655-d52d-4c95-9589-d45237fa1057\") " pod="openshift-dns/dns-default-8nb7s"
Oct 30 13:03:05.758579 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:05.758448    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-registry-tls\") pod \"image-registry-5b97bd7c47-dssmx\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") " pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:03:05.758579 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:05.758526    5093 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-service-ca-bundle podName:d5db3ba8-f1de-4fa5-9839-e7e8200da58f nodeName:}" failed. No retries permitted until 2026-10-30 13:03:09.758506522 +0000 UTC m=+8.905773744 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-service-ca-bundle") pod "router-default-668b54766c-wgk28" (UID: "d5db3ba8-f1de-4fa5-9839-e7e8200da58f") : configmap references non-existent config key: service-ca.crt
Oct 30 13:03:05.758579 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:05.758545    5093 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 13:03:05.758579 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:05.758561    5093 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b97bd7c47-dssmx: secret "image-registry-tls" not found
Oct 30 13:03:05.758579 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:05.758578    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-registry-tls\") pod \"image-registry-9677cf66b-kpdfl\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") " pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:03:05.758890 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:05.758603    5093 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-registry-tls podName:feda4324-4a5f-491b-a04c-202c222261a0 nodeName:}" failed. No retries permitted until 2026-10-30 13:03:09.758588946 +0000 UTC m=+8.905856196 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-registry-tls") pod "image-registry-5b97bd7c47-dssmx" (UID: "feda4324-4a5f-491b-a04c-202c222261a0") : secret "image-registry-tls" not found
Oct 30 13:03:05.758890 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:05.758650    5093 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 13:03:05.758890 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:05.758661    5093 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9677cf66b-kpdfl: secret "image-registry-tls" not found
Oct 30 13:03:05.758890 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:05.758687    5093 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-registry-tls podName:b4fd1eb6-625a-4649-ba27-9e8798fd1bc5 nodeName:}" failed. No retries permitted until 2026-10-30 13:03:09.758679018 +0000 UTC m=+8.905946237 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-registry-tls") pod "image-registry-9677cf66b-kpdfl" (UID: "b4fd1eb6-625a-4649-ba27-9e8798fd1bc5") : secret "image-registry-tls" not found
Oct 30 13:03:05.758890 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:05.758717    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7ceb0bb1-def2-44a5-9e31-ceffd848125d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nk2zz\" (UID: \"7ceb0bb1-def2-44a5-9e31-ceffd848125d\") " pod="openshift-insights/insights-runtime-extractor-nk2zz"
Oct 30 13:03:05.758890 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:05.758748    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a32a253a-ecaf-4f90-af1c-27a7e53b9629-metrics-certs\") pod \"network-metrics-daemon-mbncl\" (UID: \"a32a253a-ecaf-4f90-af1c-27a7e53b9629\") " pod="openshift-multus/network-metrics-daemon-mbncl"
Oct 30 13:03:05.758890 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:05.758880    5093 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 13:03:05.759176 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:05.758919    5093 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 13:03:05.759176 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:05.758923    5093 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a32a253a-ecaf-4f90-af1c-27a7e53b9629-metrics-certs podName:a32a253a-ecaf-4f90-af1c-27a7e53b9629 nodeName:}" failed. No retries permitted until 2026-10-30 13:03:09.758910506 +0000 UTC m=+8.906177738 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a32a253a-ecaf-4f90-af1c-27a7e53b9629-metrics-certs") pod "network-metrics-daemon-mbncl" (UID: "a32a253a-ecaf-4f90-af1c-27a7e53b9629") : secret "metrics-daemon-secret" not found
Oct 30 13:03:05.759176 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:05.758990    5093 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ceb0bb1-def2-44a5-9e31-ceffd848125d-insights-runtime-extractor-tls podName:7ceb0bb1-def2-44a5-9e31-ceffd848125d nodeName:}" failed. No retries permitted until 2026-10-30 13:03:09.758974137 +0000 UTC m=+8.906241363 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7ceb0bb1-def2-44a5-9e31-ceffd848125d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nk2zz" (UID: "7ceb0bb1-def2-44a5-9e31-ceffd848125d") : secret "insights-runtime-extractor-tls" not found
Oct 30 13:03:05.761215 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:05.761185    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d0b2655-d52d-4c95-9589-d45237fa1057-metrics-tls\") pod \"dns-default-8nb7s\" (UID: \"0d0b2655-d52d-4c95-9589-d45237fa1057\") " pod="openshift-dns/dns-default-8nb7s"
Oct 30 13:03:05.912579 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:05.912552    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8nb7s"
Oct 30 13:03:06.095593 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:06.095499    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x8wqt" event={"ID":"e640effb-1ff4-40f2-8d7b-53732edcd2bf","Type":"ContainerStarted","Data":"508750f5dae4702de54c972a1861464a529e58dc8331122fe1adee0a7b268972"}
Oct 30 13:03:06.216752 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:06.216720    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8nb7s"]
Oct 30 13:03:06.221011 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:06.220978    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d0b2655_d52d_4c95_9589_d45237fa1057.slice/crio-be08421414b6bc2e558f008ea8684afea1a407e73c332975fadf2b0af593f47a WatchSource:0}: Error finding container be08421414b6bc2e558f008ea8684afea1a407e73c332975fadf2b0af593f47a: Status 404 returned error can't find the container with id be08421414b6bc2e558f008ea8684afea1a407e73c332975fadf2b0af593f47a
Oct 30 13:03:06.725939 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:06.725901    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rxd99"
Oct 30 13:03:07.102663 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:07.102585    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8nb7s" event={"ID":"0d0b2655-d52d-4c95-9589-d45237fa1057","Type":"ContainerStarted","Data":"be08421414b6bc2e558f008ea8684afea1a407e73c332975fadf2b0af593f47a"}
Oct 30 13:03:08.108236 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:08.108194    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8nb7s" event={"ID":"0d0b2655-d52d-4c95-9589-d45237fa1057","Type":"ContainerStarted","Data":"a2914b04025ac70ffd91d7cc90e7cc78556494a55899cca9a8913b27bb04fdc6"}
Oct 30 13:03:08.108671 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:08.108253    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8nb7s" event={"ID":"0d0b2655-d52d-4c95-9589-d45237fa1057","Type":"ContainerStarted","Data":"14de83df02dfd8ef7a2a84a0f4024f681eb7577d419efbafc75ece8fc536fa07"}
Oct 30 13:03:08.108671 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:08.108474    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8nb7s"
Oct 30 13:03:08.982861 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:08.982831    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:08.983066 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:08.983054    5093 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 13:03:09.799628 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:09.799591    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-service-ca-bundle\") pod \"router-default-668b54766c-wgk28\" (UID: \"d5db3ba8-f1de-4fa5-9839-e7e8200da58f\") " pod="openshift-ingress/router-default-668b54766c-wgk28"
Oct 30 13:03:09.800150 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:09.799644    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-registry-tls\") pod \"image-registry-5b97bd7c47-dssmx\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") " pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:03:09.800150 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:09.799664    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-registry-tls\") pod \"image-registry-9677cf66b-kpdfl\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") " pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:03:09.800150 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:09.799712    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7ceb0bb1-def2-44a5-9e31-ceffd848125d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nk2zz\" (UID: \"7ceb0bb1-def2-44a5-9e31-ceffd848125d\") " pod="openshift-insights/insights-runtime-extractor-nk2zz"
Oct 30 13:03:09.800150 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:09.799727    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a32a253a-ecaf-4f90-af1c-27a7e53b9629-metrics-certs\") pod \"network-metrics-daemon-mbncl\" (UID: \"a32a253a-ecaf-4f90-af1c-27a7e53b9629\") " pod="openshift-multus/network-metrics-daemon-mbncl"
Oct 30 13:03:09.800386 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:09.800360    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5db3ba8-f1de-4fa5-9839-e7e8200da58f-service-ca-bundle\") pod \"router-default-668b54766c-wgk28\" (UID: \"d5db3ba8-f1de-4fa5-9839-e7e8200da58f\") " pod="openshift-ingress/router-default-668b54766c-wgk28"
Oct 30 13:03:09.801268 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:09.801248    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-668b54766c-wgk28"
Oct 30 13:03:09.802646 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:09.802616    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a32a253a-ecaf-4f90-af1c-27a7e53b9629-metrics-certs\") pod \"network-metrics-daemon-mbncl\" (UID: \"a32a253a-ecaf-4f90-af1c-27a7e53b9629\") " pod="openshift-multus/network-metrics-daemon-mbncl"
Oct 30 13:03:09.802776 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:09.802752    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7ceb0bb1-def2-44a5-9e31-ceffd848125d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nk2zz\" (UID: \"7ceb0bb1-def2-44a5-9e31-ceffd848125d\") " pod="openshift-insights/insights-runtime-extractor-nk2zz"
Oct 30 13:03:09.802815 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:09.802793    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-registry-tls\") pod \"image-registry-9677cf66b-kpdfl\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") " pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:03:09.802892 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:09.802877    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-registry-tls\") pod \"image-registry-5b97bd7c47-dssmx\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") " pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:03:09.804665 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:09.804648    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbncl"
Oct 30 13:03:09.811985 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:09.811965    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:03:09.813034 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:09.813016    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nk2zz"
Oct 30 13:03:09.977666 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:09.977633    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-6jdj4"
Oct 30 13:03:09.977817 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:09.977803    5093 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 13:03:09.981374 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:09.981357    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-6jdj4"
Oct 30 13:03:10.101853 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:10.101773    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:03:10.135863 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:10.135819    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mbncl"]
Oct 30 13:03:10.138714 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:10.138631    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nk2zz"]
Oct 30 13:03:10.139339 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:10.139306    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda32a253a_ecaf_4f90_af1c_27a7e53b9629.slice/crio-a3b36fdc756c8091f7d02202928f339e71aa1536747cc7cdda05cf42bbb7b528 WatchSource:0}: Error finding container a3b36fdc756c8091f7d02202928f339e71aa1536747cc7cdda05cf42bbb7b528: Status 404 returned error can't find the container with id a3b36fdc756c8091f7d02202928f339e71aa1536747cc7cdda05cf42bbb7b528
Oct 30 13:03:10.141941 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:10.141915    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-668b54766c-wgk28"]
Oct 30 13:03:10.143665 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:10.143270    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ceb0bb1_def2_44a5_9e31_ceffd848125d.slice/crio-88a63ffa2019c2ea16f52e4c4e7cae0aedf4b24159ca55564d88a5236632b1f9 WatchSource:0}: Error finding container 88a63ffa2019c2ea16f52e4c4e7cae0aedf4b24159ca55564d88a5236632b1f9: Status 404 returned error can't find the container with id 88a63ffa2019c2ea16f52e4c4e7cae0aedf4b24159ca55564d88a5236632b1f9
Oct 30 13:03:10.147184 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:10.147160    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5db3ba8_f1de_4fa5_9839_e7e8200da58f.slice/crio-d282cee281bffce644ae7cdc6c2070cf71398b7d2b1d0af3c398d7f3b7d7d25f WatchSource:0}: Error finding container d282cee281bffce644ae7cdc6c2070cf71398b7d2b1d0af3c398d7f3b7d7d25f: Status 404 returned error can't find the container with id d282cee281bffce644ae7cdc6c2070cf71398b7d2b1d0af3c398d7f3b7d7d25f
Oct 30 13:03:10.162048 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:10.161700    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-9677cf66b-kpdfl"]
Oct 30 13:03:10.405626 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:10.405544    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b97bd7c47-dssmx"]
Oct 30 13:03:10.408721 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:10.408692    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeda4324_4a5f_491b_a04c_202c222261a0.slice/crio-214ed0a30acf51251e023526884587395ad968bac63c7a29cb3d8916849ec008 WatchSource:0}: Error finding container 214ed0a30acf51251e023526884587395ad968bac63c7a29cb3d8916849ec008: Status 404 returned error can't find the container with id 214ed0a30acf51251e023526884587395ad968bac63c7a29cb3d8916849ec008
Oct 30 13:03:11.120353 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:11.120261    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx" event={"ID":"feda4324-4a5f-491b-a04c-202c222261a0","Type":"ContainerStarted","Data":"f14745ec62b9e61faaa2dd50e4e75f539e756d68c24b4cb85bf7e732c64bd24c"}
Oct 30 13:03:11.120353 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:11.120329    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx" event={"ID":"feda4324-4a5f-491b-a04c-202c222261a0","Type":"ContainerStarted","Data":"214ed0a30acf51251e023526884587395ad968bac63c7a29cb3d8916849ec008"}
Oct 30 13:03:11.121810 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:11.121778    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-668b54766c-wgk28" event={"ID":"d5db3ba8-f1de-4fa5-9839-e7e8200da58f","Type":"ContainerStarted","Data":"be83c86fcad7f758e50793848477d21f920b0d77e5ae6e09699a3f43597dbfea"}
Oct 30 13:03:11.121924 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:11.121829    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-668b54766c-wgk28" event={"ID":"d5db3ba8-f1de-4fa5-9839-e7e8200da58f","Type":"ContainerStarted","Data":"d282cee281bffce644ae7cdc6c2070cf71398b7d2b1d0af3c398d7f3b7d7d25f"}
Oct 30 13:03:11.123341 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:11.123313    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9677cf66b-kpdfl" event={"ID":"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5","Type":"ContainerStarted","Data":"b41393b4447073ee84bfcb4a5ab47de6e0d3fa107dbb5b18c65636cbcbe649ea"}
Oct 30 13:03:11.123447 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:11.123367    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9677cf66b-kpdfl" event={"ID":"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5","Type":"ContainerStarted","Data":"fe0c6b5557f15d4c5fb8dff51a96ba36e35616c674009dca814d51ffa6c92735"}
Oct 30 13:03:11.124576 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:11.124551    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mbncl" event={"ID":"a32a253a-ecaf-4f90-af1c-27a7e53b9629","Type":"ContainerStarted","Data":"a3b36fdc756c8091f7d02202928f339e71aa1536747cc7cdda05cf42bbb7b528"}
Oct 30 13:03:11.126290 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:11.126267    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nk2zz" event={"ID":"7ceb0bb1-def2-44a5-9e31-ceffd848125d","Type":"ContainerStarted","Data":"d7144855c9ca872dfb59597e714e997892a77a303000b7dc7847cfe16763d8d1"}
Oct 30 13:03:11.126369 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:11.126310    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nk2zz" event={"ID":"7ceb0bb1-def2-44a5-9e31-ceffd848125d","Type":"ContainerStarted","Data":"675230014f75ffb09f53f9e1755377334801b08f8291f7a8a02257ef632b5df3"}
Oct 30 13:03:11.126369 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:11.126328    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nk2zz" event={"ID":"7ceb0bb1-def2-44a5-9e31-ceffd848125d","Type":"ContainerStarted","Data":"88a63ffa2019c2ea16f52e4c4e7cae0aedf4b24159ca55564d88a5236632b1f9"}
Oct 30 13:03:11.358579 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:11.358540    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-668b54766c-wgk28"
Oct 30 13:03:11.801807 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:11.801770    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-668b54766c-wgk28"
Oct 30 13:03:11.804841 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:11.804815    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-668b54766c-wgk28"
Oct 30 13:03:12.133195 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:12.133080    5093 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 13:03:12.133195 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:12.133108    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mbncl" event={"ID":"a32a253a-ecaf-4f90-af1c-27a7e53b9629","Type":"ContainerStarted","Data":"1c7185404b8d98d4073d57bc427096911e05541eb661dfac9dca731b984ef555"}
Oct 30 13:03:12.133705 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:12.133195    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mbncl" event={"ID":"a32a253a-ecaf-4f90-af1c-27a7e53b9629","Type":"ContainerStarted","Data":"b2ac55e9095cfe42e5831526d639ac945f5ef5d397c099cbaf3f29aeec6e7991"}
Oct 30 13:03:12.133705 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:12.133097    5093 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 13:03:12.134710 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:12.134684    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-668b54766c-wgk28"
Oct 30 13:03:13.138819 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:13.138776    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nk2zz" event={"ID":"7ceb0bb1-def2-44a5-9e31-ceffd848125d","Type":"ContainerStarted","Data":"e96a7ba8d1a351777c84a7f747a3bdb03f563b783bf4d34e95a8525514dce73e"}
Oct 30 13:03:15.128750 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:15.128713    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:03:15.368540 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:15.368505    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:03:16.772543 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:16.772507    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5b97bd7c47-dssmx"]
Oct 30 13:03:18.113641 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:18.113608    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8nb7s"
Oct 30 13:03:21.165146 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:21.165102    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:03:25.372969 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:25.372938    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:03:26.665709 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:26.665672    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fthgj"]
Oct 30 13:03:26.671777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:26.671750    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fthgj"
Oct 30 13:03:26.725578 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:26.725543    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 13:03:26.725718 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:26.725582    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 13:03:26.740974 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:26.740947    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d0385014-0783-4681-a86c-1182712e0aaa-node-exporter-wtmp\") pod \"node-exporter-fthgj\" (UID: \"d0385014-0783-4681-a86c-1182712e0aaa\") " pod="openshift-monitoring/node-exporter-fthgj"
Oct 30 13:03:26.741152 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:26.740986    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0385014-0783-4681-a86c-1182712e0aaa-sys\") pod \"node-exporter-fthgj\" (UID: \"d0385014-0783-4681-a86c-1182712e0aaa\") " pod="openshift-monitoring/node-exporter-fthgj"
Oct 30 13:03:26.741152 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:26.741008    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d0385014-0783-4681-a86c-1182712e0aaa-node-exporter-tls\") pod \"node-exporter-fthgj\" (UID: \"d0385014-0783-4681-a86c-1182712e0aaa\") " pod="openshift-monitoring/node-exporter-fthgj"
Oct 30 13:03:26.741152 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:26.741125    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d0385014-0783-4681-a86c-1182712e0aaa-root\") pod \"node-exporter-fthgj\" (UID: \"d0385014-0783-4681-a86c-1182712e0aaa\") " pod="openshift-monitoring/node-exporter-fthgj"
Oct 30 13:03:26.741273 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:26.741212    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0385014-0783-4681-a86c-1182712e0aaa-metrics-client-ca\") pod \"node-exporter-fthgj\" (UID: \"d0385014-0783-4681-a86c-1182712e0aaa\") " pod="openshift-monitoring/node-exporter-fthgj"
Oct 30 13:03:26.741307 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:26.741290    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d0385014-0783-4681-a86c-1182712e0aaa-node-exporter-textfile\") pod \"node-exporter-fthgj\" (UID: \"d0385014-0783-4681-a86c-1182712e0aaa\") " pod="openshift-monitoring/node-exporter-fthgj"
Oct 30 13:03:26.741340 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:26.741322    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d0385014-0783-4681-a86c-1182712e0aaa-node-exporter-accelerators-collector-config\") pod \"node-exporter-fthgj\" (UID: \"d0385014-0783-4681-a86c-1182712e0aaa\") " pod="openshift-monitoring/node-exporter-fthgj"
Oct 30 13:03:26.741403 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:26.741385    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzd8m\" (UniqueName: \"kubernetes.io/projected/d0385014-0783-4681-a86c-1182712e0aaa-kube-api-access-zzd8m\") pod \"node-exporter-fthgj\" (UID: \"d0385014-0783-4681-a86c-1182712e0aaa\") " pod="openshift-monitoring/node-exporter-fthgj"
Oct 30 13:03:26.741439 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:26.741420    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d0385014-0783-4681-a86c-1182712e0aaa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fthgj\" (UID: \"d0385014-0783-4681-a86c-1182712e0aaa\") " pod="openshift-monitoring/node-exporter-fthgj"
Oct 30 13:03:26.774917 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:26.774889    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 13:03:26.775110 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:26.775093    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 13:03:26.775237 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:26.775179    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pbv95\""
Oct 30 13:03:26.779342 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:26.779321    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:03:26.842432 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:26.842405    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0385014-0783-4681-a86c-1182712e0aaa-sys\") pod \"node-exporter-fthgj\" (UID: \"d0385014-0783-4681-a86c-1182712e0aaa\") " pod="openshift-monitoring/node-exporter-fthgj"
Oct 30 13:03:26.842432 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:26.842434    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d0385014-0783-4681-a86c-1182712e0aaa-node-exporter-tls\") pod \"node-exporter-fthgj\" (UID: \"d0385014-0783-4681-a86c-1182712e0aaa\") " pod="openshift-monitoring/node-exporter-fthgj"
Oct 30 13:03:26.842650 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:26.842467    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d0385014-0783-4681-a86c-1182712e0aaa-root\") pod \"node-exporter-fthgj\" (UID: \"d0385014-0783-4681-a86c-1182712e0aaa\") " pod="openshift-monitoring/node-exporter-fthgj"
Oct 30 13:03:26.842650 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:26.842495    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0385014-0783-4681-a86c-1182712e0aaa-metrics-client-ca\") pod \"node-exporter-fthgj\" (UID: \"d0385014-0783-4681-a86c-1182712e0aaa\") " pod="openshift-monitoring/node-exporter-fthgj"
Oct 30 13:03:26.842650 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:26.842516    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0385014-0783-4681-a86c-1182712e0aaa-sys\") pod \"node-exporter-fthgj\" (UID: \"d0385014-0783-4681-a86c-1182712e0aaa\") " pod="openshift-monitoring/node-exporter-fthgj"
Oct 30 13:03:26.842650 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:26.842526    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d0385014-0783-4681-a86c-1182712e0aaa-node-exporter-textfile\") pod \"node-exporter-fthgj\" (UID: \"d0385014-0783-4681-a86c-1182712e0aaa\") " pod="openshift-monitoring/node-exporter-fthgj"
Oct 30 13:03:26.842650 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:26.842547    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d0385014-0783-4681-a86c-1182712e0aaa-root\") pod \"node-exporter-fthgj\" (UID: \"d0385014-0783-4681-a86c-1182712e0aaa\") " pod="openshift-monitoring/node-exporter-fthgj"
Oct 30 13:03:26.842650 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:26.842567    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d0385014-0783-4681-a86c-1182712e0aaa-node-exporter-accelerators-collector-config\") pod \"node-exporter-fthgj\" (UID: \"d0385014-0783-4681-a86c-1182712e0aaa\") " pod="openshift-monitoring/node-exporter-fthgj"
Oct 30 13:03:26.842650 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:26.842606    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzd8m\" (UniqueName: \"kubernetes.io/projected/d0385014-0783-4681-a86c-1182712e0aaa-kube-api-access-zzd8m\") pod \"node-exporter-fthgj\" (UID: \"d0385014-0783-4681-a86c-1182712e0aaa\") " pod="openshift-monitoring/node-exporter-fthgj"
Oct 30 13:03:26.842650 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:26.842641    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d0385014-0783-4681-a86c-1182712e0aaa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fthgj\" (UID: \"d0385014-0783-4681-a86c-1182712e0aaa\") " pod="openshift-monitoring/node-exporter-fthgj"
Oct 30 13:03:26.843024 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:26.842700    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d0385014-0783-4681-a86c-1182712e0aaa-node-exporter-wtmp\") pod \"node-exporter-fthgj\" (UID: \"d0385014-0783-4681-a86c-1182712e0aaa\") " pod="openshift-monitoring/node-exporter-fthgj"
Oct 30 13:03:26.843024 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:26.842860    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d0385014-0783-4681-a86c-1182712e0aaa-node-exporter-wtmp\") pod \"node-exporter-fthgj\" (UID: \"d0385014-0783-4681-a86c-1182712e0aaa\") " pod="openshift-monitoring/node-exporter-fthgj"
Oct 30 13:03:26.843024 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:26.842889    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d0385014-0783-4681-a86c-1182712e0aaa-node-exporter-textfile\") pod \"node-exporter-fthgj\" (UID: \"d0385014-0783-4681-a86c-1182712e0aaa\") " pod="openshift-monitoring/node-exporter-fthgj"
Oct 30 13:03:26.843278 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:26.843257    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0385014-0783-4681-a86c-1182712e0aaa-metrics-client-ca\") pod \"node-exporter-fthgj\" (UID: \"d0385014-0783-4681-a86c-1182712e0aaa\") " pod="openshift-monitoring/node-exporter-fthgj"
Oct 30 13:03:26.843330 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:26.843305    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d0385014-0783-4681-a86c-1182712e0aaa-node-exporter-accelerators-collector-config\") pod \"node-exporter-fthgj\" (UID: \"d0385014-0783-4681-a86c-1182712e0aaa\") " pod="openshift-monitoring/node-exporter-fthgj"
Oct 30 13:03:26.844949 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:26.844931    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d0385014-0783-4681-a86c-1182712e0aaa-node-exporter-tls\") pod \"node-exporter-fthgj\" (UID: \"d0385014-0783-4681-a86c-1182712e0aaa\") " pod="openshift-monitoring/node-exporter-fthgj"
Oct 30 13:03:26.845041 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:26.845002    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d0385014-0783-4681-a86c-1182712e0aaa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fthgj\" (UID: \"d0385014-0783-4681-a86c-1182712e0aaa\") " pod="openshift-monitoring/node-exporter-fthgj"
Oct 30 13:03:26.863900 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:26.863873    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzd8m\" (UniqueName: \"kubernetes.io/projected/d0385014-0783-4681-a86c-1182712e0aaa-kube-api-access-zzd8m\") pod \"node-exporter-fthgj\" (UID: \"d0385014-0783-4681-a86c-1182712e0aaa\") " pod="openshift-monitoring/node-exporter-fthgj"
Oct 30 13:03:26.988755 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:26.988719    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fthgj"
Oct 30 13:03:27.003458 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:03:27.003429    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0385014_0783_4681_a86c_1182712e0aaa.slice/crio-094a7c731118243381341128c501732ac793bf3e72a8c8a43bebd3fb6f74bcee WatchSource:0}: Error finding container 094a7c731118243381341128c501732ac793bf3e72a8c8a43bebd3fb6f74bcee: Status 404 returned error can't find the container with id 094a7c731118243381341128c501732ac793bf3e72a8c8a43bebd3fb6f74bcee
Oct 30 13:03:27.195556 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:27.195515    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fthgj" event={"ID":"d0385014-0783-4681-a86c-1182712e0aaa","Type":"ContainerStarted","Data":"094a7c731118243381341128c501732ac793bf3e72a8c8a43bebd3fb6f74bcee"}
Oct 30 13:03:28.849898 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:28.849823    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-67tnm"
Oct 30 13:03:28.950416 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:28.950386    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wpqvv"
Oct 30 13:03:29.206258 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:29.206210    5093 generic.go:356] "Generic (PLEG): container finished" podID="d0385014-0783-4681-a86c-1182712e0aaa" containerID="b41dfaab4e7faefd911a06b236a0a7faa15ac6e4bd6371aa798aaf350fe645c7" exitCode=0
Oct 30 13:03:29.206450 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:29.206294    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fthgj" event={"ID":"d0385014-0783-4681-a86c-1182712e0aaa","Type":"ContainerDied","Data":"b41dfaab4e7faefd911a06b236a0a7faa15ac6e4bd6371aa798aaf350fe645c7"}
Oct 30 13:03:30.212482 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:30.212449    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fthgj" event={"ID":"d0385014-0783-4681-a86c-1182712e0aaa","Type":"ContainerStarted","Data":"6212856ca323290c5c7a5d186f91e6606761b4bfe282c08d945652ef3fe32ab2"}
Oct 30 13:03:30.212879 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:30.212502    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fthgj" event={"ID":"d0385014-0783-4681-a86c-1182712e0aaa","Type":"ContainerStarted","Data":"99c80638c800ce06446c0fc7367a65cd116effb9bf665a50ea619455094181dd"}
Oct 30 13:03:30.281589 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:30.281533    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fthgj" podStartSLOduration=2.778886839 podStartE2EDuration="4.281519377s" totalImagesPullingTime="1.425878211s" totalInitContainerRuntime="76.754327ms" isStatefulPod=true podCreationTimestamp="2026-10-30 13:03:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:30.267770064 +0000 UTC m=+29.415037304" watchObservedRunningTime="2026-10-30 13:03:30.281519377 +0000 UTC m=+29.428786677"
Oct 30 13:03:36.728797 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:36.728764    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rxd99"
Oct 30 13:03:40.131200 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:40.131159    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-9677cf66b-kpdfl"]
Oct 30 13:03:41.793706 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:41.793597    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx" podUID="feda4324-4a5f-491b-a04c-202c222261a0" containerName="registry" containerID="cri-o://f14745ec62b9e61faaa2dd50e4e75f539e756d68c24b4cb85bf7e732c64bd24c" gracePeriod=30
Oct 30 13:03:42.040579 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:42.040551    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:03:42.178714 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:42.178684    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/feda4324-4a5f-491b-a04c-202c222261a0-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/feda4324-4a5f-491b-a04c-202c222261a0-image-registry-private-configuration\") pod \"feda4324-4a5f-491b-a04c-202c222261a0\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") "
Oct 30 13:03:42.178882 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:42.178738    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/feda4324-4a5f-491b-a04c-202c222261a0-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/feda4324-4a5f-491b-a04c-202c222261a0-trusted-ca\") pod \"feda4324-4a5f-491b-a04c-202c222261a0\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") "
Oct 30 13:03:42.178882 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:42.178788    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feda4324-4a5f-491b-a04c-202c222261a0-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/feda4324-4a5f-491b-a04c-202c222261a0-ca-trust-extracted\") pod \"feda4324-4a5f-491b-a04c-202c222261a0\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") "
Oct 30 13:03:42.178882 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:42.178830    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-kube-api-access-ncqh7\" (UniqueName: \"kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-kube-api-access-ncqh7\") pod \"feda4324-4a5f-491b-a04c-202c222261a0\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") "
Oct 30 13:03:42.179040 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:42.178881    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/feda4324-4a5f-491b-a04c-202c222261a0-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/feda4324-4a5f-491b-a04c-202c222261a0-registry-certificates\") pod \"feda4324-4a5f-491b-a04c-202c222261a0\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") "
Oct 30 13:03:42.179040 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:42.178912    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/feda4324-4a5f-491b-a04c-202c222261a0-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/feda4324-4a5f-491b-a04c-202c222261a0-installation-pull-secrets\") pod \"feda4324-4a5f-491b-a04c-202c222261a0\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") "
Oct 30 13:03:42.179040 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:42.178946    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-registry-tls\" (UniqueName: \"kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-registry-tls\") pod \"feda4324-4a5f-491b-a04c-202c222261a0\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") "
Oct 30 13:03:42.179040 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:42.178993    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-bound-sa-token\") pod \"feda4324-4a5f-491b-a04c-202c222261a0\" (UID: \"feda4324-4a5f-491b-a04c-202c222261a0\") "
Oct 30 13:03:42.179403 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:42.179333    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feda4324-4a5f-491b-a04c-202c222261a0-trusted-ca" pod "feda4324-4a5f-491b-a04c-202c222261a0" (UID: "feda4324-4a5f-491b-a04c-202c222261a0"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:03:42.179498 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:42.179445    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feda4324-4a5f-491b-a04c-202c222261a0-registry-certificates" pod "feda4324-4a5f-491b-a04c-202c222261a0" (UID: "feda4324-4a5f-491b-a04c-202c222261a0"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:03:42.181543 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:42.181483    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-registry-tls" pod "feda4324-4a5f-491b-a04c-202c222261a0" (UID: "feda4324-4a5f-491b-a04c-202c222261a0"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:03:42.181665 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:42.181546    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feda4324-4a5f-491b-a04c-202c222261a0-installation-pull-secrets" pod "feda4324-4a5f-491b-a04c-202c222261a0" (UID: "feda4324-4a5f-491b-a04c-202c222261a0"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:03:42.181665 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:42.181622    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feda4324-4a5f-491b-a04c-202c222261a0-image-registry-private-configuration" pod "feda4324-4a5f-491b-a04c-202c222261a0" (UID: "feda4324-4a5f-491b-a04c-202c222261a0"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:03:42.181873 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:42.181849    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-kube-api-access-ncqh7" pod "feda4324-4a5f-491b-a04c-202c222261a0" (UID: "feda4324-4a5f-491b-a04c-202c222261a0"). InnerVolumeSpecName "kube-api-access-ncqh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:03:42.181971 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:42.181932    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-bound-sa-token" pod "feda4324-4a5f-491b-a04c-202c222261a0" (UID: "feda4324-4a5f-491b-a04c-202c222261a0"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:03:42.186188 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:42.186119    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feda4324-4a5f-491b-a04c-202c222261a0-ca-trust-extracted" pod "feda4324-4a5f-491b-a04c-202c222261a0" (UID: "feda4324-4a5f-491b-a04c-202c222261a0"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:03:42.260735 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:42.260696    5093 generic.go:356] "Generic (PLEG): container finished" podID="feda4324-4a5f-491b-a04c-202c222261a0" containerID="f14745ec62b9e61faaa2dd50e4e75f539e756d68c24b4cb85bf7e732c64bd24c" exitCode=0
Oct 30 13:03:42.260897 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:42.260752    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx"
Oct 30 13:03:42.260897 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:42.260780    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx" event={"ID":"feda4324-4a5f-491b-a04c-202c222261a0","Type":"ContainerDied","Data":"f14745ec62b9e61faaa2dd50e4e75f539e756d68c24b4cb85bf7e732c64bd24c"}
Oct 30 13:03:42.260897 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:42.260856    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b97bd7c47-dssmx" event={"ID":"feda4324-4a5f-491b-a04c-202c222261a0","Type":"ContainerDied","Data":"214ed0a30acf51251e023526884587395ad968bac63c7a29cb3d8916849ec008"}
Oct 30 13:03:42.260897 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:42.260880    5093 scope.go:122] "RemoveContainer" containerID="f14745ec62b9e61faaa2dd50e4e75f539e756d68c24b4cb85bf7e732c64bd24c"
Oct 30 13:03:42.269579 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:42.269560    5093 scope.go:122] "RemoveContainer" containerID="f14745ec62b9e61faaa2dd50e4e75f539e756d68c24b4cb85bf7e732c64bd24c"
Oct 30 13:03:42.269821 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:42.269802    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f14745ec62b9e61faaa2dd50e4e75f539e756d68c24b4cb85bf7e732c64bd24c\": container with ID starting with f14745ec62b9e61faaa2dd50e4e75f539e756d68c24b4cb85bf7e732c64bd24c not found: ID does not exist" containerID="f14745ec62b9e61faaa2dd50e4e75f539e756d68c24b4cb85bf7e732c64bd24c"
Oct 30 13:03:42.269887 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:42.269830    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f14745ec62b9e61faaa2dd50e4e75f539e756d68c24b4cb85bf7e732c64bd24c"} err="failed to get container status \"f14745ec62b9e61faaa2dd50e4e75f539e756d68c24b4cb85bf7e732c64bd24c\": rpc error: code = NotFound desc = could not find container \"f14745ec62b9e61faaa2dd50e4e75f539e756d68c24b4cb85bf7e732c64bd24c\": container with ID starting with f14745ec62b9e61faaa2dd50e4e75f539e756d68c24b4cb85bf7e732c64bd24c not found: ID does not exist"
Oct 30 13:03:42.280237 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:42.280217    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncqh7\" (UniqueName: \"kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-kube-api-access-ncqh7\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:03:42.280237 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:42.280238    5093 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/feda4324-4a5f-491b-a04c-202c222261a0-registry-certificates\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:03:42.280370 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:42.280248    5093 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/feda4324-4a5f-491b-a04c-202c222261a0-installation-pull-secrets\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:03:42.280370 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:42.280258    5093 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-registry-tls\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:03:42.280370 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:42.280268    5093 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/feda4324-4a5f-491b-a04c-202c222261a0-bound-sa-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:03:42.280370 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:42.280277    5093 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/feda4324-4a5f-491b-a04c-202c222261a0-image-registry-private-configuration\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:03:42.280370 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:42.280285    5093 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/feda4324-4a5f-491b-a04c-202c222261a0-trusted-ca\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:03:42.280370 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:42.280297    5093 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/feda4324-4a5f-491b-a04c-202c222261a0-ca-trust-extracted\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:03:42.315546 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:42.315515    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5b97bd7c47-dssmx"]
Oct 30 13:03:42.320214 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:42.320191    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5b97bd7c47-dssmx"]
Oct 30 13:03:42.996825 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:03:42.996787    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="feda4324-4a5f-491b-a04c-202c222261a0" path="/var/lib/kubelet/pods/feda4324-4a5f-491b-a04c-202c222261a0/volumes"
Oct 30 13:03:51.165670 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:03:51.165633    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:04:05.151709 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:05.151647    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-9677cf66b-kpdfl" podUID="b4fd1eb6-625a-4649-ba27-9e8798fd1bc5" containerName="registry" containerID="cri-o://b41393b4447073ee84bfcb4a5ab47de6e0d3fa107dbb5b18c65636cbcbe649ea" gracePeriod=30
Oct 30 13:04:05.351613 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:05.351580    5093 generic.go:356] "Generic (PLEG): container finished" podID="b4fd1eb6-625a-4649-ba27-9e8798fd1bc5" containerID="b41393b4447073ee84bfcb4a5ab47de6e0d3fa107dbb5b18c65636cbcbe649ea" exitCode=0
Oct 30 13:04:05.351771 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:05.351651    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9677cf66b-kpdfl" event={"ID":"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5","Type":"ContainerDied","Data":"b41393b4447073ee84bfcb4a5ab47de6e0d3fa107dbb5b18c65636cbcbe649ea"}
Oct 30 13:04:05.399089 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:05.399064    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:04:05.465073 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:05.465042    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-ca-trust-extracted\") pod \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") "
Oct 30 13:04:05.465269 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:05.465085    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-kube-api-access-vdrpm\" (UniqueName: \"kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-kube-api-access-vdrpm\") pod \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") "
Oct 30 13:04:05.465269 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:05.465139    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-registry-tls\" (UniqueName: \"kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-registry-tls\") pod \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") "
Oct 30 13:04:05.465483 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:05.465448    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-trusted-ca\") pod \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") "
Oct 30 13:04:05.465810 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:05.465770    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-trusted-ca" pod "b4fd1eb6-625a-4649-ba27-9e8798fd1bc5" (UID: "b4fd1eb6-625a-4649-ba27-9e8798fd1bc5"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:04:05.468040 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:05.468010    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-registry-tls" pod "b4fd1eb6-625a-4649-ba27-9e8798fd1bc5" (UID: "b4fd1eb6-625a-4649-ba27-9e8798fd1bc5"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:04:05.468196 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:05.468162    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-kube-api-access-vdrpm" pod "b4fd1eb6-625a-4649-ba27-9e8798fd1bc5" (UID: "b4fd1eb6-625a-4649-ba27-9e8798fd1bc5"). InnerVolumeSpecName "kube-api-access-vdrpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:04:05.472937 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:05.472909    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-ca-trust-extracted" pod "b4fd1eb6-625a-4649-ba27-9e8798fd1bc5" (UID: "b4fd1eb6-625a-4649-ba27-9e8798fd1bc5"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:04:05.566198 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:05.566159    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-image-registry-private-configuration\") pod \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") "
Oct 30 13:04:05.566198 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:05.566200    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-bound-sa-token\") pod \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") "
Oct 30 13:04:05.566414 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:05.566226    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-installation-pull-secrets\") pod \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") "
Oct 30 13:04:05.566414 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:05.566266    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-registry-certificates\") pod \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\" (UID: \"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5\") "
Oct 30 13:04:05.566478 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:05.566450    5093 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-ca-trust-extracted\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:04:05.566478 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:05.566469    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdrpm\" (UniqueName: \"kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-kube-api-access-vdrpm\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:04:05.566552 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:05.566484    5093 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-registry-tls\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:04:05.566552 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:05.566499    5093 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-trusted-ca\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:04:05.566718 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:05.566697    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-registry-certificates" pod "b4fd1eb6-625a-4649-ba27-9e8798fd1bc5" (UID: "b4fd1eb6-625a-4649-ba27-9e8798fd1bc5"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:04:05.568486 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:05.568449    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-image-registry-private-configuration" pod "b4fd1eb6-625a-4649-ba27-9e8798fd1bc5" (UID: "b4fd1eb6-625a-4649-ba27-9e8798fd1bc5"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:04:05.568617 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:05.568536    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-installation-pull-secrets" pod "b4fd1eb6-625a-4649-ba27-9e8798fd1bc5" (UID: "b4fd1eb6-625a-4649-ba27-9e8798fd1bc5"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:04:05.568617 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:05.568535    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-bound-sa-token" pod "b4fd1eb6-625a-4649-ba27-9e8798fd1bc5" (UID: "b4fd1eb6-625a-4649-ba27-9e8798fd1bc5"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:04:05.667447 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:05.667407    5093 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-registry-certificates\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:04:05.667447 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:05.667440    5093 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-image-registry-private-configuration\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:04:05.667447 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:05.667454    5093 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-bound-sa-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:04:05.667670 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:05.667467    5093 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5-installation-pull-secrets\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:04:06.357262 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:06.357226    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9677cf66b-kpdfl" event={"ID":"b4fd1eb6-625a-4649-ba27-9e8798fd1bc5","Type":"ContainerDied","Data":"fe0c6b5557f15d4c5fb8dff51a96ba36e35616c674009dca814d51ffa6c92735"}
Oct 30 13:04:06.357712 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:06.357267    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9677cf66b-kpdfl"
Oct 30 13:04:06.357712 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:06.357286    5093 scope.go:122] "RemoveContainer" containerID="b41393b4447073ee84bfcb4a5ab47de6e0d3fa107dbb5b18c65636cbcbe649ea"
Oct 30 13:04:06.424291 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:06.424257    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-9677cf66b-kpdfl"]
Oct 30 13:04:06.437564 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:06.437529    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-9677cf66b-kpdfl"]
Oct 30 13:04:06.996719 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:06.996686    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4fd1eb6-625a-4649-ba27-9e8798fd1bc5" path="/var/lib/kubelet/pods/b4fd1eb6-625a-4649-ba27-9e8798fd1bc5/volumes"
Oct 30 13:04:10.370349 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:10.370309    5093 patch_prober.go:29] interesting pod/image-registry-9677cf66b-kpdfl container/registry namespace/openshift-image-registry: Readiness probe status=failure output="Get \"https://10.132.0.18:5000/healthz\": context deadline exceeded" start-of-body=
Oct 30 13:04:10.370814 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:10.370379    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-9677cf66b-kpdfl" podUID="b4fd1eb6-625a-4649-ba27-9e8798fd1bc5" containerName="registry" probeResult="failure" output="Get \"https://10.132.0.18:5000/healthz\": context deadline exceeded"
Oct 30 13:04:12.385071 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:12.385039    5093 generic.go:356] "Generic (PLEG): container finished" podID="79664154-5454-4294-baad-12bdfad6fe68" containerID="64af381430f07be2de9fc933146d3337b68583cf06e1405ad9f25c9aa87670bb" exitCode=0
Oct 30 13:04:12.385497 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:12.385112    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fhnzt" event={"ID":"79664154-5454-4294-baad-12bdfad6fe68","Type":"ContainerDied","Data":"64af381430f07be2de9fc933146d3337b68583cf06e1405ad9f25c9aa87670bb"}
Oct 30 13:04:12.385497 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:12.385487    5093 scope.go:122] "RemoveContainer" containerID="64af381430f07be2de9fc933146d3337b68583cf06e1405ad9f25c9aa87670bb"
Oct 30 13:04:13.391467 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:13.391431    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fhnzt" event={"ID":"79664154-5454-4294-baad-12bdfad6fe68","Type":"ContainerStarted","Data":"319abc9ab817642bccc93b80fa9121aefd7c37bc4df2c6fdca151010b3e9d92c"}
Oct 30 13:04:19.417122 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:19.417085    5093 generic.go:356] "Generic (PLEG): container finished" podID="f4acf40a-d5b0-46e2-b87f-3b96fc9678c4" containerID="88919911d03e7a59c8ca765bf094fd152dbb01b64488b1c70c627d30991354ed" exitCode=0
Oct 30 13:04:19.417523 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:19.417161    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-9wlmq-z26dn" event={"ID":"f4acf40a-d5b0-46e2-b87f-3b96fc9678c4","Type":"ContainerDied","Data":"88919911d03e7a59c8ca765bf094fd152dbb01b64488b1c70c627d30991354ed"}
Oct 30 13:04:20.542401 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:20.542372    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-9wlmq-z26dn"
Oct 30 13:04:20.562044 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:20.562010    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4acf40a-d5b0-46e2-b87f-3b96fc9678c4-kube-api-access-xdh8w\" (UniqueName: \"kubernetes.io/projected/f4acf40a-d5b0-46e2-b87f-3b96fc9678c4-kube-api-access-xdh8w\") pod \"f4acf40a-d5b0-46e2-b87f-3b96fc9678c4\" (UID: \"f4acf40a-d5b0-46e2-b87f-3b96fc9678c4\") "
Oct 30 13:04:20.562239 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:20.562099    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4acf40a-d5b0-46e2-b87f-3b96fc9678c4-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4acf40a-d5b0-46e2-b87f-3b96fc9678c4-trusted-ca-bundle\") pod \"f4acf40a-d5b0-46e2-b87f-3b96fc9678c4\" (UID: \"f4acf40a-d5b0-46e2-b87f-3b96fc9678c4\") "
Oct 30 13:04:20.562239 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:20.562187    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4acf40a-d5b0-46e2-b87f-3b96fc9678c4-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f4acf40a-d5b0-46e2-b87f-3b96fc9678c4-archives-path\") pod \"f4acf40a-d5b0-46e2-b87f-3b96fc9678c4\" (UID: \"f4acf40a-d5b0-46e2-b87f-3b96fc9678c4\") "
Oct 30 13:04:20.562239 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:20.562221    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4acf40a-d5b0-46e2-b87f-3b96fc9678c4-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4acf40a-d5b0-46e2-b87f-3b96fc9678c4-service-ca-bundle\") pod \"f4acf40a-d5b0-46e2-b87f-3b96fc9678c4\" (UID: \"f4acf40a-d5b0-46e2-b87f-3b96fc9678c4\") "
Oct 30 13:04:20.562555 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:20.562525    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4acf40a-d5b0-46e2-b87f-3b96fc9678c4-archives-path" pod "f4acf40a-d5b0-46e2-b87f-3b96fc9678c4" (UID: "f4acf40a-d5b0-46e2-b87f-3b96fc9678c4"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:04:20.562625 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:20.562532    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4acf40a-d5b0-46e2-b87f-3b96fc9678c4-trusted-ca-bundle" pod "f4acf40a-d5b0-46e2-b87f-3b96fc9678c4" (UID: "f4acf40a-d5b0-46e2-b87f-3b96fc9678c4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:04:20.562625 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:20.562571    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4acf40a-d5b0-46e2-b87f-3b96fc9678c4-service-ca-bundle" pod "f4acf40a-d5b0-46e2-b87f-3b96fc9678c4" (UID: "f4acf40a-d5b0-46e2-b87f-3b96fc9678c4"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:04:20.564315 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:20.564287    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4acf40a-d5b0-46e2-b87f-3b96fc9678c4-kube-api-access-xdh8w" pod "f4acf40a-d5b0-46e2-b87f-3b96fc9678c4" (UID: "f4acf40a-d5b0-46e2-b87f-3b96fc9678c4"). InnerVolumeSpecName "kube-api-access-xdh8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:04:20.662870 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:20.662838    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdh8w\" (UniqueName: \"kubernetes.io/projected/f4acf40a-d5b0-46e2-b87f-3b96fc9678c4-kube-api-access-xdh8w\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:04:20.662870 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:20.662866    5093 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4acf40a-d5b0-46e2-b87f-3b96fc9678c4-trusted-ca-bundle\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:04:20.662870 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:20.662877    5093 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f4acf40a-d5b0-46e2-b87f-3b96fc9678c4-archives-path\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:04:20.663122 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:20.662888    5093 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4acf40a-d5b0-46e2-b87f-3b96fc9678c4-service-ca-bundle\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:04:21.165316 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:04:21.165267    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:04:21.425568 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:21.425540    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-9wlmq-z26dn"
Oct 30 13:04:21.425713 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:21.425537    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-9wlmq-z26dn" event={"ID":"f4acf40a-d5b0-46e2-b87f-3b96fc9678c4","Type":"ContainerDied","Data":"3cc6fafce08f8f3d202026631327bb8e095f97ff186665b5b1996bd0fe1fbc6c"}
Oct 30 13:04:21.425713 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:21.425665    5093 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cc6fafce08f8f3d202026631327bb8e095f97ff186665b5b1996bd0fe1fbc6c"
Oct 30 13:04:26.445148 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:26.445107    5093 generic.go:356] "Generic (PLEG): container finished" podID="155bbdd4-b566-467f-88c6-ec5b4b881a09" containerID="2718d9cd1ed17a33c335f8c10461bb03121ca28199b5424a91e71e2b0303f8f9" exitCode=0
Oct 30 13:04:26.445545 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:26.445186    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-lgssx" event={"ID":"155bbdd4-b566-467f-88c6-ec5b4b881a09","Type":"ContainerDied","Data":"2718d9cd1ed17a33c335f8c10461bb03121ca28199b5424a91e71e2b0303f8f9"}
Oct 30 13:04:26.445593 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:26.445553    5093 scope.go:122] "RemoveContainer" containerID="2718d9cd1ed17a33c335f8c10461bb03121ca28199b5424a91e71e2b0303f8f9"
Oct 30 13:04:27.127244 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:27.127203    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-f2hn7-xz8zj"]
Oct 30 13:04:27.127587 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:27.127568    5093 cpu_manager.go:414] "containerMap: removing container" podUID="f4acf40a-d5b0-46e2-b87f-3b96fc9678c4" containerName="insights-gathering"
Oct 30 13:04:27.127673 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:27.127589    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4acf40a-d5b0-46e2-b87f-3b96fc9678c4" containerName="insights-gathering"
Oct 30 13:04:27.127673 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:27.127607    5093 cpu_manager.go:414] "containerMap: removing container" podUID="feda4324-4a5f-491b-a04c-202c222261a0" containerName="registry"
Oct 30 13:04:27.127673 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:27.127615    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feda4324-4a5f-491b-a04c-202c222261a0" containerName="registry"
Oct 30 13:04:27.127673 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:27.127653    5093 cpu_manager.go:414] "containerMap: removing container" podUID="b4fd1eb6-625a-4649-ba27-9e8798fd1bc5" containerName="registry"
Oct 30 13:04:27.127673 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:27.127662    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4fd1eb6-625a-4649-ba27-9e8798fd1bc5" containerName="registry"
Oct 30 13:04:27.127915 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:27.127742    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="feda4324-4a5f-491b-a04c-202c222261a0" containerName="registry"
Oct 30 13:04:27.127915 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:27.127759    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4fd1eb6-625a-4649-ba27-9e8798fd1bc5" containerName="registry"
Oct 30 13:04:27.127915 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:27.127771    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4acf40a-d5b0-46e2-b87f-3b96fc9678c4" containerName="insights-gathering"
Oct 30 13:04:27.130808 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:27.130784    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-f2hn7-xz8zj"
Oct 30 13:04:27.147715 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:27.147686    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-f2hn7-xz8zj"]
Oct 30 13:04:27.203623 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:27.203586    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0af6efa4-62fa-402f-926e-3a142a0a28a1-service-ca-bundle\") pod \"periodic-gathering-f2hn7-xz8zj\" (UID: \"0af6efa4-62fa-402f-926e-3a142a0a28a1\") " pod="openshift-insights/periodic-gathering-f2hn7-xz8zj"
Oct 30 13:04:27.203623 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:27.203625    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0af6efa4-62fa-402f-926e-3a142a0a28a1-trusted-ca-bundle\") pod \"periodic-gathering-f2hn7-xz8zj\" (UID: \"0af6efa4-62fa-402f-926e-3a142a0a28a1\") " pod="openshift-insights/periodic-gathering-f2hn7-xz8zj"
Oct 30 13:04:27.203845 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:27.203747    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsskc\" (UniqueName: \"kubernetes.io/projected/0af6efa4-62fa-402f-926e-3a142a0a28a1-kube-api-access-vsskc\") pod \"periodic-gathering-f2hn7-xz8zj\" (UID: \"0af6efa4-62fa-402f-926e-3a142a0a28a1\") " pod="openshift-insights/periodic-gathering-f2hn7-xz8zj"
Oct 30 13:04:27.203845 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:27.203802    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/0af6efa4-62fa-402f-926e-3a142a0a28a1-archives-path\") pod \"periodic-gathering-f2hn7-xz8zj\" (UID: \"0af6efa4-62fa-402f-926e-3a142a0a28a1\") " pod="openshift-insights/periodic-gathering-f2hn7-xz8zj"
Oct 30 13:04:27.304763 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:27.304724    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0af6efa4-62fa-402f-926e-3a142a0a28a1-trusted-ca-bundle\") pod \"periodic-gathering-f2hn7-xz8zj\" (UID: \"0af6efa4-62fa-402f-926e-3a142a0a28a1\") " pod="openshift-insights/periodic-gathering-f2hn7-xz8zj"
Oct 30 13:04:27.304896 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:27.304792    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsskc\" (UniqueName: \"kubernetes.io/projected/0af6efa4-62fa-402f-926e-3a142a0a28a1-kube-api-access-vsskc\") pod \"periodic-gathering-f2hn7-xz8zj\" (UID: \"0af6efa4-62fa-402f-926e-3a142a0a28a1\") " pod="openshift-insights/periodic-gathering-f2hn7-xz8zj"
Oct 30 13:04:27.304896 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:27.304852    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/0af6efa4-62fa-402f-926e-3a142a0a28a1-archives-path\") pod \"periodic-gathering-f2hn7-xz8zj\" (UID: \"0af6efa4-62fa-402f-926e-3a142a0a28a1\") " pod="openshift-insights/periodic-gathering-f2hn7-xz8zj"
Oct 30 13:04:27.305015 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:27.304986    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0af6efa4-62fa-402f-926e-3a142a0a28a1-service-ca-bundle\") pod \"periodic-gathering-f2hn7-xz8zj\" (UID: \"0af6efa4-62fa-402f-926e-3a142a0a28a1\") " pod="openshift-insights/periodic-gathering-f2hn7-xz8zj"
Oct 30 13:04:27.305332 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:27.305308    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/0af6efa4-62fa-402f-926e-3a142a0a28a1-archives-path\") pod \"periodic-gathering-f2hn7-xz8zj\" (UID: \"0af6efa4-62fa-402f-926e-3a142a0a28a1\") " pod="openshift-insights/periodic-gathering-f2hn7-xz8zj"
Oct 30 13:04:27.305592 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:27.305573    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0af6efa4-62fa-402f-926e-3a142a0a28a1-service-ca-bundle\") pod \"periodic-gathering-f2hn7-xz8zj\" (UID: \"0af6efa4-62fa-402f-926e-3a142a0a28a1\") " pod="openshift-insights/periodic-gathering-f2hn7-xz8zj"
Oct 30 13:04:27.305629 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:27.305588    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0af6efa4-62fa-402f-926e-3a142a0a28a1-trusted-ca-bundle\") pod \"periodic-gathering-f2hn7-xz8zj\" (UID: \"0af6efa4-62fa-402f-926e-3a142a0a28a1\") " pod="openshift-insights/periodic-gathering-f2hn7-xz8zj"
Oct 30 13:04:27.325730 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:27.325700    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsskc\" (UniqueName: \"kubernetes.io/projected/0af6efa4-62fa-402f-926e-3a142a0a28a1-kube-api-access-vsskc\") pod \"periodic-gathering-f2hn7-xz8zj\" (UID: \"0af6efa4-62fa-402f-926e-3a142a0a28a1\") " pod="openshift-insights/periodic-gathering-f2hn7-xz8zj"
Oct 30 13:04:27.441597 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:27.441564    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-f2hn7-xz8zj"
Oct 30 13:04:27.451981 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:27.451950    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-lgssx" event={"ID":"155bbdd4-b566-467f-88c6-ec5b4b881a09","Type":"ContainerStarted","Data":"590b167dddf5e1f0e3c46ae98322616374a854aa3e27922b3c49c2141ad3c5f0"}
Oct 30 13:04:27.732209 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:27.732112    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-f2hn7-xz8zj"]
Oct 30 13:04:27.735301 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:04:27.735265    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0af6efa4_62fa_402f_926e_3a142a0a28a1.slice/crio-2f55515b5d57af017bb732fb1146a8819232fcc580d3f2f08f10aa61c9c27233 WatchSource:0}: Error finding container 2f55515b5d57af017bb732fb1146a8819232fcc580d3f2f08f10aa61c9c27233: Status 404 returned error can't find the container with id 2f55515b5d57af017bb732fb1146a8819232fcc580d3f2f08f10aa61c9c27233
Oct 30 13:04:28.457194 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:28.457159    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-f2hn7-xz8zj" event={"ID":"0af6efa4-62fa-402f-926e-3a142a0a28a1","Type":"ContainerStarted","Data":"ea02aef6be11706654f6f950253fd17d3c3e63a1de37d4ebb6a64489308f120f"}
Oct 30 13:04:28.457585 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:28.457207    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-f2hn7-xz8zj" event={"ID":"0af6efa4-62fa-402f-926e-3a142a0a28a1","Type":"ContainerStarted","Data":"2f55515b5d57af017bb732fb1146a8819232fcc580d3f2f08f10aa61c9c27233"}
Oct 30 13:04:28.498115 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:28.498061    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-f2hn7-xz8zj" podStartSLOduration=1.498047026 podStartE2EDuration="1.498047026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:28.496771681 +0000 UTC m=+87.644038922" watchObservedRunningTime="2026-10-30 13:04:28.498047026 +0000 UTC m=+87.645314261"
Oct 30 13:04:32.479150 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:32.479095    5093 generic.go:356] "Generic (PLEG): container finished" podID="5cbda802-f819-4ea6-846e-440cea4afed2" containerID="0ec814bf9af33ffe20a9f56154cad2671a9939f1a592b0099e108dbf4e8fa5f7" exitCode=0
Oct 30 13:04:32.479647 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:32.479173    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lt2qg" event={"ID":"5cbda802-f819-4ea6-846e-440cea4afed2","Type":"ContainerDied","Data":"0ec814bf9af33ffe20a9f56154cad2671a9939f1a592b0099e108dbf4e8fa5f7"}
Oct 30 13:04:32.479647 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:32.479640    5093 scope.go:122] "RemoveContainer" containerID="0ec814bf9af33ffe20a9f56154cad2671a9939f1a592b0099e108dbf4e8fa5f7"
Oct 30 13:04:33.485951 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:04:33.485920    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lt2qg" event={"ID":"5cbda802-f819-4ea6-846e-440cea4afed2","Type":"ContainerStarted","Data":"4d5d2abb832ba4c865553614ce446e1ff9957b2925ba01dba88e56bb831bda4c"}
Oct 30 13:04:51.165629 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:04:51.165566    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:05:21.164842 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:05:21.164804    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:05:51.166426 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:05:51.166389    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:05:51.801485 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:05:51.801451    5093 generic.go:356] "Generic (PLEG): container finished" podID="0af6efa4-62fa-402f-926e-3a142a0a28a1" containerID="ea02aef6be11706654f6f950253fd17d3c3e63a1de37d4ebb6a64489308f120f" exitCode=0
Oct 30 13:05:51.801651 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:05:51.801496    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-f2hn7-xz8zj" event={"ID":"0af6efa4-62fa-402f-926e-3a142a0a28a1","Type":"ContainerDied","Data":"ea02aef6be11706654f6f950253fd17d3c3e63a1de37d4ebb6a64489308f120f"}
Oct 30 13:05:52.930010 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:05:52.929985    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-f2hn7-xz8zj"
Oct 30 13:05:52.988791 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:05:52.988747    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0af6efa4-62fa-402f-926e-3a142a0a28a1-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0af6efa4-62fa-402f-926e-3a142a0a28a1-service-ca-bundle\") pod \"0af6efa4-62fa-402f-926e-3a142a0a28a1\" (UID: \"0af6efa4-62fa-402f-926e-3a142a0a28a1\") "
Oct 30 13:05:52.989000 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:05:52.988819    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0af6efa4-62fa-402f-926e-3a142a0a28a1-kube-api-access-vsskc\" (UniqueName: \"kubernetes.io/projected/0af6efa4-62fa-402f-926e-3a142a0a28a1-kube-api-access-vsskc\") pod \"0af6efa4-62fa-402f-926e-3a142a0a28a1\" (UID: \"0af6efa4-62fa-402f-926e-3a142a0a28a1\") "
Oct 30 13:05:52.989000 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:05:52.988873    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0af6efa4-62fa-402f-926e-3a142a0a28a1-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0af6efa4-62fa-402f-926e-3a142a0a28a1-trusted-ca-bundle\") pod \"0af6efa4-62fa-402f-926e-3a142a0a28a1\" (UID: \"0af6efa4-62fa-402f-926e-3a142a0a28a1\") "
Oct 30 13:05:52.989000 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:05:52.988912    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0af6efa4-62fa-402f-926e-3a142a0a28a1-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/0af6efa4-62fa-402f-926e-3a142a0a28a1-archives-path\") pod \"0af6efa4-62fa-402f-926e-3a142a0a28a1\" (UID: \"0af6efa4-62fa-402f-926e-3a142a0a28a1\") "
Oct 30 13:05:52.989246 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:05:52.989210    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0af6efa4-62fa-402f-926e-3a142a0a28a1-service-ca-bundle" pod "0af6efa4-62fa-402f-926e-3a142a0a28a1" (UID: "0af6efa4-62fa-402f-926e-3a142a0a28a1"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:05:52.989376 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:05:52.989273    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0af6efa4-62fa-402f-926e-3a142a0a28a1-trusted-ca-bundle" pod "0af6efa4-62fa-402f-926e-3a142a0a28a1" (UID: "0af6efa4-62fa-402f-926e-3a142a0a28a1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:05:52.989376 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:05:52.989296    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0af6efa4-62fa-402f-926e-3a142a0a28a1-archives-path" pod "0af6efa4-62fa-402f-926e-3a142a0a28a1" (UID: "0af6efa4-62fa-402f-926e-3a142a0a28a1"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:52.990955 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:05:52.990929    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0af6efa4-62fa-402f-926e-3a142a0a28a1-kube-api-access-vsskc" pod "0af6efa4-62fa-402f-926e-3a142a0a28a1" (UID: "0af6efa4-62fa-402f-926e-3a142a0a28a1"). InnerVolumeSpecName "kube-api-access-vsskc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:53.089619 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:05:53.089545    5093 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0af6efa4-62fa-402f-926e-3a142a0a28a1-trusted-ca-bundle\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:05:53.089619 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:05:53.089571    5093 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/0af6efa4-62fa-402f-926e-3a142a0a28a1-archives-path\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:05:53.089619 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:05:53.089581    5093 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0af6efa4-62fa-402f-926e-3a142a0a28a1-service-ca-bundle\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:05:53.089619 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:05:53.089591    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsskc\" (UniqueName: \"kubernetes.io/projected/0af6efa4-62fa-402f-926e-3a142a0a28a1-kube-api-access-vsskc\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:05:53.810486 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:05:53.810447    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-f2hn7-xz8zj" event={"ID":"0af6efa4-62fa-402f-926e-3a142a0a28a1","Type":"ContainerDied","Data":"2f55515b5d57af017bb732fb1146a8819232fcc580d3f2f08f10aa61c9c27233"}
Oct 30 13:05:53.810486 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:05:53.810481    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-f2hn7-xz8zj"
Oct 30 13:05:53.810754 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:05:53.810497    5093 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f55515b5d57af017bb732fb1146a8819232fcc580d3f2f08f10aa61c9c27233"
Oct 30 13:06:06.124948 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:06.124906    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqdscd"]
Oct 30 13:06:06.125443 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:06.125359    5093 cpu_manager.go:414] "containerMap: removing container" podUID="0af6efa4-62fa-402f-926e-3a142a0a28a1" containerName="insights-gathering"
Oct 30 13:06:06.125443 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:06.125379    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0af6efa4-62fa-402f-926e-3a142a0a28a1" containerName="insights-gathering"
Oct 30 13:06:06.125519 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:06.125477    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="0af6efa4-62fa-402f-926e-3a142a0a28a1" containerName="insights-gathering"
Oct 30 13:06:06.128924 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:06.128906    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqdscd"
Oct 30 13:06:06.147624 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:06.147593    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 13:06:06.161148 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:06.161098    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 13:06:06.161299 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:06.161153    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-hcc29\""
Oct 30 13:06:06.162147 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:06.162106    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqdscd"]
Oct 30 13:06:06.183006 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:06.182977    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6c77c3a6-b076-4cf7-811d-6176b6425e06-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqdscd\" (UID: \"6c77c3a6-b076-4cf7-811d-6176b6425e06\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqdscd"
Oct 30 13:06:06.183189 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:06.183017    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6c77c3a6-b076-4cf7-811d-6176b6425e06-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqdscd\" (UID: \"6c77c3a6-b076-4cf7-811d-6176b6425e06\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqdscd"
Oct 30 13:06:06.183189 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:06.183105    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqxkd\" (UniqueName: \"kubernetes.io/projected/6c77c3a6-b076-4cf7-811d-6176b6425e06-kube-api-access-dqxkd\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqdscd\" (UID: \"6c77c3a6-b076-4cf7-811d-6176b6425e06\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqdscd"
Oct 30 13:06:06.284060 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:06.284017    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqxkd\" (UniqueName: \"kubernetes.io/projected/6c77c3a6-b076-4cf7-811d-6176b6425e06-kube-api-access-dqxkd\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqdscd\" (UID: \"6c77c3a6-b076-4cf7-811d-6176b6425e06\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqdscd"
Oct 30 13:06:06.284373 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:06.284089    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6c77c3a6-b076-4cf7-811d-6176b6425e06-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqdscd\" (UID: \"6c77c3a6-b076-4cf7-811d-6176b6425e06\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqdscd"
Oct 30 13:06:06.284373 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:06.284163    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6c77c3a6-b076-4cf7-811d-6176b6425e06-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqdscd\" (UID: \"6c77c3a6-b076-4cf7-811d-6176b6425e06\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqdscd"
Oct 30 13:06:06.284537 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:06.284509    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6c77c3a6-b076-4cf7-811d-6176b6425e06-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqdscd\" (UID: \"6c77c3a6-b076-4cf7-811d-6176b6425e06\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqdscd"
Oct 30 13:06:06.284575 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:06.284527    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6c77c3a6-b076-4cf7-811d-6176b6425e06-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqdscd\" (UID: \"6c77c3a6-b076-4cf7-811d-6176b6425e06\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqdscd"
Oct 30 13:06:06.308279 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:06.308247    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqxkd\" (UniqueName: \"kubernetes.io/projected/6c77c3a6-b076-4cf7-811d-6176b6425e06-kube-api-access-dqxkd\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqdscd\" (UID: \"6c77c3a6-b076-4cf7-811d-6176b6425e06\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqdscd"
Oct 30 13:06:06.444308 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:06.444272    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqdscd"
Oct 30 13:06:06.739585 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:06.739435    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqdscd"]
Oct 30 13:06:06.742244 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:06:06.742211    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c77c3a6_b076_4cf7_811d_6176b6425e06.slice/crio-cb28d7a5fe157ab8180d74636731b4633dd3d3e1fb8ebbbb3901e793c89ef0f6 WatchSource:0}: Error finding container cb28d7a5fe157ab8180d74636731b4633dd3d3e1fb8ebbbb3901e793c89ef0f6: Status 404 returned error can't find the container with id cb28d7a5fe157ab8180d74636731b4633dd3d3e1fb8ebbbb3901e793c89ef0f6
Oct 30 13:06:06.867246 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:06.867205    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqdscd" event={"ID":"6c77c3a6-b076-4cf7-811d-6176b6425e06","Type":"ContainerStarted","Data":"cb28d7a5fe157ab8180d74636731b4633dd3d3e1fb8ebbbb3901e793c89ef0f6"}
Oct 30 13:06:07.733605 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:07.733566    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcn7c"]
Oct 30 13:06:07.737425 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:07.737397    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcn7c"
Oct 30 13:06:07.757211 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:07.757173    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcn7c"]
Oct 30 13:06:07.769353 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:07.769325    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-n8rkm\""
Oct 30 13:06:07.798970 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:07.798902    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f3e51708-e19e-424a-8e37-09817a673212-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcn7c\" (UID: \"f3e51708-e19e-424a-8e37-09817a673212\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcn7c"
Oct 30 13:06:07.799188 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:07.799066    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f3e51708-e19e-424a-8e37-09817a673212-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcn7c\" (UID: \"f3e51708-e19e-424a-8e37-09817a673212\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcn7c"
Oct 30 13:06:07.799188 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:07.799095    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv4qr\" (UniqueName: \"kubernetes.io/projected/f3e51708-e19e-424a-8e37-09817a673212-kube-api-access-cv4qr\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcn7c\" (UID: \"f3e51708-e19e-424a-8e37-09817a673212\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcn7c"
Oct 30 13:06:07.893344 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:07.893296    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77d5fw"]
Oct 30 13:06:07.896903 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:07.896881    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77d5fw"
Oct 30 13:06:07.900379 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:07.900355    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f3e51708-e19e-424a-8e37-09817a673212-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcn7c\" (UID: \"f3e51708-e19e-424a-8e37-09817a673212\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcn7c"
Oct 30 13:06:07.900520 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:07.900397    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv4qr\" (UniqueName: \"kubernetes.io/projected/f3e51708-e19e-424a-8e37-09817a673212-kube-api-access-cv4qr\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcn7c\" (UID: \"f3e51708-e19e-424a-8e37-09817a673212\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcn7c"
Oct 30 13:06:07.900520 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:07.900459    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f3e51708-e19e-424a-8e37-09817a673212-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcn7c\" (UID: \"f3e51708-e19e-424a-8e37-09817a673212\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcn7c"
Oct 30 13:06:07.900816 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:07.900789    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f3e51708-e19e-424a-8e37-09817a673212-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcn7c\" (UID: \"f3e51708-e19e-424a-8e37-09817a673212\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcn7c"
Oct 30 13:06:07.901014 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:07.900988    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f3e51708-e19e-424a-8e37-09817a673212-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcn7c\" (UID: \"f3e51708-e19e-424a-8e37-09817a673212\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcn7c"
Oct 30 13:06:07.914444 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:07.914419    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-xx5xw\""
Oct 30 13:06:07.919420 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:07.919392    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77d5fw"]
Oct 30 13:06:07.926861 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:07.926799    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv4qr\" (UniqueName: \"kubernetes.io/projected/f3e51708-e19e-424a-8e37-09817a673212-kube-api-access-cv4qr\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcn7c\" (UID: \"f3e51708-e19e-424a-8e37-09817a673212\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcn7c"
Oct 30 13:06:07.995060 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:07.994975    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agp879"]
Oct 30 13:06:07.998836 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:07.998811    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agp879"
Oct 30 13:06:08.001698 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.001666    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2979d0d-8de1-42cf-b174-2dfb76f0bfa6-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77d5fw\" (UID: \"a2979d0d-8de1-42cf-b174-2dfb76f0bfa6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77d5fw"
Oct 30 13:06:08.001824 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.001726    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2979d0d-8de1-42cf-b174-2dfb76f0bfa6-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77d5fw\" (UID: \"a2979d0d-8de1-42cf-b174-2dfb76f0bfa6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77d5fw"
Oct 30 13:06:08.001824 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.001795    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsbct\" (UniqueName: \"kubernetes.io/projected/a2979d0d-8de1-42cf-b174-2dfb76f0bfa6-kube-api-access-jsbct\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77d5fw\" (UID: \"a2979d0d-8de1-42cf-b174-2dfb76f0bfa6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77d5fw"
Oct 30 13:06:08.015175 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.015143    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-9djrh\""
Oct 30 13:06:08.020876 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.020844    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agp879"]
Oct 30 13:06:08.056896 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.056868    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcn7c"
Oct 30 13:06:08.102548 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.102509    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2979d0d-8de1-42cf-b174-2dfb76f0bfa6-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77d5fw\" (UID: \"a2979d0d-8de1-42cf-b174-2dfb76f0bfa6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77d5fw"
Oct 30 13:06:08.102548 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.102554    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsbct\" (UniqueName: \"kubernetes.io/projected/a2979d0d-8de1-42cf-b174-2dfb76f0bfa6-kube-api-access-jsbct\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77d5fw\" (UID: \"a2979d0d-8de1-42cf-b174-2dfb76f0bfa6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77d5fw"
Oct 30 13:06:08.102787 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.102606    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83d191b4-087e-49cb-b73c-2aa61b5a2cbc-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agp879\" (UID: \"83d191b4-087e-49cb-b73c-2aa61b5a2cbc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agp879"
Oct 30 13:06:08.102787 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.102663    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83d191b4-087e-49cb-b73c-2aa61b5a2cbc-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agp879\" (UID: \"83d191b4-087e-49cb-b73c-2aa61b5a2cbc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agp879"
Oct 30 13:06:08.102787 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.102685    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4bwq\" (UniqueName: \"kubernetes.io/projected/83d191b4-087e-49cb-b73c-2aa61b5a2cbc-kube-api-access-d4bwq\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agp879\" (UID: \"83d191b4-087e-49cb-b73c-2aa61b5a2cbc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agp879"
Oct 30 13:06:08.102787 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.102711    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2979d0d-8de1-42cf-b174-2dfb76f0bfa6-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77d5fw\" (UID: \"a2979d0d-8de1-42cf-b174-2dfb76f0bfa6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77d5fw"
Oct 30 13:06:08.102978 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.102957    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2979d0d-8de1-42cf-b174-2dfb76f0bfa6-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77d5fw\" (UID: \"a2979d0d-8de1-42cf-b174-2dfb76f0bfa6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77d5fw"
Oct 30 13:06:08.103075 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.103043    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2979d0d-8de1-42cf-b174-2dfb76f0bfa6-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77d5fw\" (UID: \"a2979d0d-8de1-42cf-b174-2dfb76f0bfa6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77d5fw"
Oct 30 13:06:08.129416 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.129383    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsbct\" (UniqueName: \"kubernetes.io/projected/a2979d0d-8de1-42cf-b174-2dfb76f0bfa6-kube-api-access-jsbct\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77d5fw\" (UID: \"a2979d0d-8de1-42cf-b174-2dfb76f0bfa6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77d5fw"
Oct 30 13:06:08.204107 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.204062    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83d191b4-087e-49cb-b73c-2aa61b5a2cbc-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agp879\" (UID: \"83d191b4-087e-49cb-b73c-2aa61b5a2cbc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agp879"
Oct 30 13:06:08.204107 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.204107    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4bwq\" (UniqueName: \"kubernetes.io/projected/83d191b4-087e-49cb-b73c-2aa61b5a2cbc-kube-api-access-d4bwq\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agp879\" (UID: \"83d191b4-087e-49cb-b73c-2aa61b5a2cbc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agp879"
Oct 30 13:06:08.204353 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.204191    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83d191b4-087e-49cb-b73c-2aa61b5a2cbc-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agp879\" (UID: \"83d191b4-087e-49cb-b73c-2aa61b5a2cbc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agp879"
Oct 30 13:06:08.204573 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.204552    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83d191b4-087e-49cb-b73c-2aa61b5a2cbc-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agp879\" (UID: \"83d191b4-087e-49cb-b73c-2aa61b5a2cbc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agp879"
Oct 30 13:06:08.204641 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.204590    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83d191b4-087e-49cb-b73c-2aa61b5a2cbc-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agp879\" (UID: \"83d191b4-087e-49cb-b73c-2aa61b5a2cbc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agp879"
Oct 30 13:06:08.215050 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.215017    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77d5fw"
Oct 30 13:06:08.226540 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.226511    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4bwq\" (UniqueName: \"kubernetes.io/projected/83d191b4-087e-49cb-b73c-2aa61b5a2cbc-kube-api-access-d4bwq\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agp879\" (UID: \"83d191b4-087e-49cb-b73c-2aa61b5a2cbc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agp879"
Oct 30 13:06:08.300557 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.300522    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs"]
Oct 30 13:06:08.305508 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.305473    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs"
Oct 30 13:06:08.318825 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.318795    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agp879"
Oct 30 13:06:08.329523 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.325523    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 13:06:08.329523 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.327639    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs"]
Oct 30 13:06:08.369021 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.368943    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcn7c"]
Oct 30 13:06:08.373337 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:06:08.373299    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3e51708_e19e_424a_8e37_09817a673212.slice/crio-7ec81e1879f2c9d3a5bcb4b2b9e15616ca26ffe60e4838466832bed890a9c00c WatchSource:0}: Error finding container 7ec81e1879f2c9d3a5bcb4b2b9e15616ca26ffe60e4838466832bed890a9c00c: Status 404 returned error can't find the container with id 7ec81e1879f2c9d3a5bcb4b2b9e15616ca26ffe60e4838466832bed890a9c00c
Oct 30 13:06:08.382285 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.382234    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-cw6t5\""
Oct 30 13:06:08.405555 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.405517    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c417d75a-c5bf-42fa-8aff-335a63c1295e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-knngs\" (UID: \"c417d75a-c5bf-42fa-8aff-335a63c1295e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs"
Oct 30 13:06:08.405717 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.405577    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjf2s\" (UniqueName: \"kubernetes.io/projected/c417d75a-c5bf-42fa-8aff-335a63c1295e-kube-api-access-hjf2s\") pod \"maas-default-gateway-openshift-default-67bf986f99-knngs\" (UID: \"c417d75a-c5bf-42fa-8aff-335a63c1295e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs"
Oct 30 13:06:08.405717 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.405615    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c417d75a-c5bf-42fa-8aff-335a63c1295e-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-knngs\" (UID: \"c417d75a-c5bf-42fa-8aff-335a63c1295e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs"
Oct 30 13:06:08.405717 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.405641    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c417d75a-c5bf-42fa-8aff-335a63c1295e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-knngs\" (UID: \"c417d75a-c5bf-42fa-8aff-335a63c1295e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs"
Oct 30 13:06:08.405717 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.405664    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c417d75a-c5bf-42fa-8aff-335a63c1295e-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-knngs\" (UID: \"c417d75a-c5bf-42fa-8aff-335a63c1295e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs"
Oct 30 13:06:08.405717 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.405699    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c417d75a-c5bf-42fa-8aff-335a63c1295e-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-knngs\" (UID: \"c417d75a-c5bf-42fa-8aff-335a63c1295e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs"
Oct 30 13:06:08.405927 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.405728    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c417d75a-c5bf-42fa-8aff-335a63c1295e-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-knngs\" (UID: \"c417d75a-c5bf-42fa-8aff-335a63c1295e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs"
Oct 30 13:06:08.405927 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.405755    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c417d75a-c5bf-42fa-8aff-335a63c1295e-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-knngs\" (UID: \"c417d75a-c5bf-42fa-8aff-335a63c1295e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs"
Oct 30 13:06:08.405927 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.405786    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c417d75a-c5bf-42fa-8aff-335a63c1295e-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-knngs\" (UID: \"c417d75a-c5bf-42fa-8aff-335a63c1295e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs"
Oct 30 13:06:08.506843 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.506808    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c417d75a-c5bf-42fa-8aff-335a63c1295e-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-knngs\" (UID: \"c417d75a-c5bf-42fa-8aff-335a63c1295e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs"
Oct 30 13:06:08.507034 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.506868    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c417d75a-c5bf-42fa-8aff-335a63c1295e-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-knngs\" (UID: \"c417d75a-c5bf-42fa-8aff-335a63c1295e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs"
Oct 30 13:06:08.507034 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.506904    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c417d75a-c5bf-42fa-8aff-335a63c1295e-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-knngs\" (UID: \"c417d75a-c5bf-42fa-8aff-335a63c1295e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs"
Oct 30 13:06:08.507034 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.506925    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c417d75a-c5bf-42fa-8aff-335a63c1295e-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-knngs\" (UID: \"c417d75a-c5bf-42fa-8aff-335a63c1295e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs"
Oct 30 13:06:08.507034 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.506956    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c417d75a-c5bf-42fa-8aff-335a63c1295e-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-knngs\" (UID: \"c417d75a-c5bf-42fa-8aff-335a63c1295e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs"
Oct 30 13:06:08.507034 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.506978    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c417d75a-c5bf-42fa-8aff-335a63c1295e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-knngs\" (UID: \"c417d75a-c5bf-42fa-8aff-335a63c1295e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs"
Oct 30 13:06:08.507034 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.507023    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjf2s\" (UniqueName: \"kubernetes.io/projected/c417d75a-c5bf-42fa-8aff-335a63c1295e-kube-api-access-hjf2s\") pod \"maas-default-gateway-openshift-default-67bf986f99-knngs\" (UID: \"c417d75a-c5bf-42fa-8aff-335a63c1295e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs"
Oct 30 13:06:08.507369 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.507071    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c417d75a-c5bf-42fa-8aff-335a63c1295e-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-knngs\" (UID: \"c417d75a-c5bf-42fa-8aff-335a63c1295e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs"
Oct 30 13:06:08.507369 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.507123    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c417d75a-c5bf-42fa-8aff-335a63c1295e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-knngs\" (UID: \"c417d75a-c5bf-42fa-8aff-335a63c1295e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs"
Oct 30 13:06:08.507856 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.507819    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c417d75a-c5bf-42fa-8aff-335a63c1295e-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-knngs\" (UID: \"c417d75a-c5bf-42fa-8aff-335a63c1295e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs"
Oct 30 13:06:08.507984 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.507874    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c417d75a-c5bf-42fa-8aff-335a63c1295e-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-knngs\" (UID: \"c417d75a-c5bf-42fa-8aff-335a63c1295e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs"
Oct 30 13:06:08.508037 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.507981    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c417d75a-c5bf-42fa-8aff-335a63c1295e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-knngs\" (UID: \"c417d75a-c5bf-42fa-8aff-335a63c1295e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs"
Oct 30 13:06:08.508153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.508118    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c417d75a-c5bf-42fa-8aff-335a63c1295e-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-knngs\" (UID: \"c417d75a-c5bf-42fa-8aff-335a63c1295e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs"
Oct 30 13:06:08.508269 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.508249    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c417d75a-c5bf-42fa-8aff-335a63c1295e-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-knngs\" (UID: \"c417d75a-c5bf-42fa-8aff-335a63c1295e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs"
Oct 30 13:06:08.510356 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.510329    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c417d75a-c5bf-42fa-8aff-335a63c1295e-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-knngs\" (UID: \"c417d75a-c5bf-42fa-8aff-335a63c1295e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs"
Oct 30 13:06:08.510504 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.510485    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c417d75a-c5bf-42fa-8aff-335a63c1295e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-knngs\" (UID: \"c417d75a-c5bf-42fa-8aff-335a63c1295e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs"
Oct 30 13:06:08.519556 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.519495    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77d5fw"]
Oct 30 13:06:08.523648 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:06:08.523622    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2979d0d_8de1_42cf_b174_2dfb76f0bfa6.slice/crio-f0b640397adc10de083fed3837bc9787c453fcae00c15e2aab0f431afa3860f7 WatchSource:0}: Error finding container f0b640397adc10de083fed3837bc9787c453fcae00c15e2aab0f431afa3860f7: Status 404 returned error can't find the container with id f0b640397adc10de083fed3837bc9787c453fcae00c15e2aab0f431afa3860f7
Oct 30 13:06:08.528448 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.528425    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c417d75a-c5bf-42fa-8aff-335a63c1295e-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-knngs\" (UID: \"c417d75a-c5bf-42fa-8aff-335a63c1295e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs"
Oct 30 13:06:08.528551 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.528532    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjf2s\" (UniqueName: \"kubernetes.io/projected/c417d75a-c5bf-42fa-8aff-335a63c1295e-kube-api-access-hjf2s\") pod \"maas-default-gateway-openshift-default-67bf986f99-knngs\" (UID: \"c417d75a-c5bf-42fa-8aff-335a63c1295e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs"
Oct 30 13:06:08.627716 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.627556    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agp879"]
Oct 30 13:06:08.631600 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:06:08.631569    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83d191b4_087e_49cb_b73c_2aa61b5a2cbc.slice/crio-6398625cabff59ebd6349a5a5fedca2915f74ab833daaacc6a918a243d52ed99 WatchSource:0}: Error finding container 6398625cabff59ebd6349a5a5fedca2915f74ab833daaacc6a918a243d52ed99: Status 404 returned error can't find the container with id 6398625cabff59ebd6349a5a5fedca2915f74ab833daaacc6a918a243d52ed99
Oct 30 13:06:08.633632 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.633604    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs"
Oct 30 13:06:08.878919 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.878823    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcn7c" event={"ID":"f3e51708-e19e-424a-8e37-09817a673212","Type":"ContainerStarted","Data":"7ec81e1879f2c9d3a5bcb4b2b9e15616ca26ffe60e4838466832bed890a9c00c"}
Oct 30 13:06:08.880730 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.880692    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agp879" event={"ID":"83d191b4-087e-49cb-b73c-2aa61b5a2cbc","Type":"ContainerStarted","Data":"6398625cabff59ebd6349a5a5fedca2915f74ab833daaacc6a918a243d52ed99"}
Oct 30 13:06:08.882514 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.882485    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77d5fw" event={"ID":"a2979d0d-8de1-42cf-b174-2dfb76f0bfa6","Type":"ContainerStarted","Data":"f0b640397adc10de083fed3837bc9787c453fcae00c15e2aab0f431afa3860f7"}
Oct 30 13:06:08.957383 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:08.957350    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs"]
Oct 30 13:06:08.962008 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:06:08.961968    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc417d75a_c5bf_42fa_8aff_335a63c1295e.slice/crio-99226e58d1d627da6e0464fcad298ea1921e083fa49f00e38ef30aa0b55d6d19 WatchSource:0}: Error finding container 99226e58d1d627da6e0464fcad298ea1921e083fa49f00e38ef30aa0b55d6d19: Status 404 returned error can't find the container with id 99226e58d1d627da6e0464fcad298ea1921e083fa49f00e38ef30aa0b55d6d19
Oct 30 13:06:09.502095 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:09.499985    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449mzfd"]
Oct 30 13:06:09.504888 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:09.504223    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449mzfd"
Oct 30 13:06:09.534676 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:09.534643    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-7q7n9\""
Oct 30 13:06:09.536453 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:09.536428    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449mzfd"]
Oct 30 13:06:09.617820 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:09.617778    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e42e185-65bd-44f7-a864-96695117283b-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449mzfd\" (UID: \"0e42e185-65bd-44f7-a864-96695117283b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449mzfd"
Oct 30 13:06:09.618025 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:09.617834    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e42e185-65bd-44f7-a864-96695117283b-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449mzfd\" (UID: \"0e42e185-65bd-44f7-a864-96695117283b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449mzfd"
Oct 30 13:06:09.618025 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:09.617870    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzlzf\" (UniqueName: \"kubernetes.io/projected/0e42e185-65bd-44f7-a864-96695117283b-kube-api-access-mzlzf\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449mzfd\" (UID: \"0e42e185-65bd-44f7-a864-96695117283b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449mzfd"
Oct 30 13:06:09.719374 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:09.719333    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e42e185-65bd-44f7-a864-96695117283b-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449mzfd\" (UID: \"0e42e185-65bd-44f7-a864-96695117283b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449mzfd"
Oct 30 13:06:09.719570 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:09.719389    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e42e185-65bd-44f7-a864-96695117283b-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449mzfd\" (UID: \"0e42e185-65bd-44f7-a864-96695117283b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449mzfd"
Oct 30 13:06:09.719570 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:09.719423    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzlzf\" (UniqueName: \"kubernetes.io/projected/0e42e185-65bd-44f7-a864-96695117283b-kube-api-access-mzlzf\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449mzfd\" (UID: \"0e42e185-65bd-44f7-a864-96695117283b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449mzfd"
Oct 30 13:06:09.720249 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:09.720221    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e42e185-65bd-44f7-a864-96695117283b-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449mzfd\" (UID: \"0e42e185-65bd-44f7-a864-96695117283b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449mzfd"
Oct 30 13:06:09.720490 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:09.720469    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e42e185-65bd-44f7-a864-96695117283b-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449mzfd\" (UID: \"0e42e185-65bd-44f7-a864-96695117283b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449mzfd"
Oct 30 13:06:09.742036 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:09.741996    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzlzf\" (UniqueName: \"kubernetes.io/projected/0e42e185-65bd-44f7-a864-96695117283b-kube-api-access-mzlzf\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449mzfd\" (UID: \"0e42e185-65bd-44f7-a864-96695117283b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449mzfd"
Oct 30 13:06:09.823388 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:09.823309    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d8mtd"]
Oct 30 13:06:09.832951 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:09.832912    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449mzfd"
Oct 30 13:06:09.844791 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:09.844741    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d8mtd"]
Oct 30 13:06:09.845280 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:09.845108    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d8mtd"
Oct 30 13:06:09.862979 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:09.862928    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-6928j\""
Oct 30 13:06:09.899956 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:09.899887    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs" event={"ID":"c417d75a-c5bf-42fa-8aff-335a63c1295e","Type":"ContainerStarted","Data":"99226e58d1d627da6e0464fcad298ea1921e083fa49f00e38ef30aa0b55d6d19"}
Oct 30 13:06:09.927635 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:09.926050    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8h72\" (UniqueName: \"kubernetes.io/projected/0e5821d2-c302-4393-8943-72cdff7e41ad-kube-api-access-d8h72\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d8mtd\" (UID: \"0e5821d2-c302-4393-8943-72cdff7e41ad\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d8mtd"
Oct 30 13:06:09.927635 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:09.926179    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e5821d2-c302-4393-8943-72cdff7e41ad-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d8mtd\" (UID: \"0e5821d2-c302-4393-8943-72cdff7e41ad\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d8mtd"
Oct 30 13:06:09.927635 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:09.926244    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e5821d2-c302-4393-8943-72cdff7e41ad-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d8mtd\" (UID: \"0e5821d2-c302-4393-8943-72cdff7e41ad\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d8mtd"
Oct 30 13:06:10.027395 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:10.027358    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8h72\" (UniqueName: \"kubernetes.io/projected/0e5821d2-c302-4393-8943-72cdff7e41ad-kube-api-access-d8h72\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d8mtd\" (UID: \"0e5821d2-c302-4393-8943-72cdff7e41ad\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d8mtd"
Oct 30 13:06:10.027597 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:10.027437    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e5821d2-c302-4393-8943-72cdff7e41ad-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d8mtd\" (UID: \"0e5821d2-c302-4393-8943-72cdff7e41ad\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d8mtd"
Oct 30 13:06:10.027597 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:10.027550    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e5821d2-c302-4393-8943-72cdff7e41ad-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d8mtd\" (UID: \"0e5821d2-c302-4393-8943-72cdff7e41ad\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d8mtd"
Oct 30 13:06:10.028019 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:10.027997    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e5821d2-c302-4393-8943-72cdff7e41ad-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d8mtd\" (UID: \"0e5821d2-c302-4393-8943-72cdff7e41ad\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d8mtd"
Oct 30 13:06:10.028551 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:10.028525    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e5821d2-c302-4393-8943-72cdff7e41ad-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d8mtd\" (UID: \"0e5821d2-c302-4393-8943-72cdff7e41ad\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d8mtd"
Oct 30 13:06:10.050469 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:10.050437    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8h72\" (UniqueName: \"kubernetes.io/projected/0e5821d2-c302-4393-8943-72cdff7e41ad-kube-api-access-d8h72\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d8mtd\" (UID: \"0e5821d2-c302-4393-8943-72cdff7e41ad\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d8mtd"
Oct 30 13:06:10.160465 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:10.160383    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449mzfd"]
Oct 30 13:06:10.164424 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:06:10.164390    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e42e185_65bd_44f7_a864_96695117283b.slice/crio-e78d056dc516129670d51c4d307cc8e8ac80ae0623d521f616a7f818cd1c45a0 WatchSource:0}: Error finding container e78d056dc516129670d51c4d307cc8e8ac80ae0623d521f616a7f818cd1c45a0: Status 404 returned error can't find the container with id e78d056dc516129670d51c4d307cc8e8ac80ae0623d521f616a7f818cd1c45a0
Oct 30 13:06:10.197019 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:10.196977    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d8mtd"
Oct 30 13:06:10.553773 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:10.553740    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d8mtd"]
Oct 30 13:06:10.561327 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:06:10.561247    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e5821d2_c302_4393_8943_72cdff7e41ad.slice/crio-e5abb99dd96e1ad2a9017d3f3be0025078e8209f2aff2ac84fd847324c73dc44 WatchSource:0}: Error finding container e5abb99dd96e1ad2a9017d3f3be0025078e8209f2aff2ac84fd847324c73dc44: Status 404 returned error can't find the container with id e5abb99dd96e1ad2a9017d3f3be0025078e8209f2aff2ac84fd847324c73dc44
Oct 30 13:06:10.914330 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:10.914187    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d8mtd" event={"ID":"0e5821d2-c302-4393-8943-72cdff7e41ad","Type":"ContainerStarted","Data":"e5abb99dd96e1ad2a9017d3f3be0025078e8209f2aff2ac84fd847324c73dc44"}
Oct 30 13:06:10.917036 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:10.916951    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449mzfd" event={"ID":"0e42e185-65bd-44f7-a864-96695117283b","Type":"ContainerStarted","Data":"e78d056dc516129670d51c4d307cc8e8ac80ae0623d521f616a7f818cd1c45a0"}
Oct 30 13:06:14.739241 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:14.739203    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:06:14.739648 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:14.739312    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:06:14.739648 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:14.739367    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:06:14.951789 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:14.951748    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs" event={"ID":"c417d75a-c5bf-42fa-8aff-335a63c1295e","Type":"ContainerStarted","Data":"ebce1b79b819e2495a9dc6584f37315e5b653389f7a936ae939e2c46153e4daf"}
Oct 30 13:06:15.013052 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:15.012901    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs" podStartSLOduration=1.239379808 podStartE2EDuration="7.012881821s" totalImagesPullingTime="5.773502013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:15.011077081 +0000 UTC m=+194.158344324" watchObservedRunningTime="2026-10-30 13:06:15.012881821 +0000 UTC m=+194.160149063"
Oct 30 13:06:15.634387 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:15.634352    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs"
Oct 30 13:06:15.640473 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:15.640444    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs"
Oct 30 13:06:15.959427 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:15.959362    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs"
Oct 30 13:06:15.961382 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:15.961123    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs"
Oct 30 13:06:17.976529 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:17.974995    5093 generic.go:356] "Generic (PLEG): container finished" podID="6c77c3a6-b076-4cf7-811d-6176b6425e06" containerID="c1cea9caa1f208f429d5f31bfc079d5d768700e2177da6bac033e4b624b13bc4" exitCode=0
Oct 30 13:06:17.976529 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:17.976237    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqdscd" event={"ID":"6c77c3a6-b076-4cf7-811d-6176b6425e06","Type":"ContainerDied","Data":"c1cea9caa1f208f429d5f31bfc079d5d768700e2177da6bac033e4b624b13bc4"}
Oct 30 13:06:18.987921 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:18.987876    5093 generic.go:356] "Generic (PLEG): container finished" podID="0e5821d2-c302-4393-8943-72cdff7e41ad" containerID="fae897135361009c5e5c5f7cf2a6363e08ffd1e10e9fbd320c532dd20eb65774" exitCode=0
Oct 30 13:06:18.988391 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:18.987983    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d8mtd" event={"ID":"0e5821d2-c302-4393-8943-72cdff7e41ad","Type":"ContainerDied","Data":"fae897135361009c5e5c5f7cf2a6363e08ffd1e10e9fbd320c532dd20eb65774"}
Oct 30 13:06:18.990700 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:18.990673    5093 generic.go:356] "Generic (PLEG): container finished" podID="0e42e185-65bd-44f7-a864-96695117283b" containerID="82f6997d5321836c9a3f715c941d845d686ff243bf80466e2923727e74cf4bd4" exitCode=0
Oct 30 13:06:18.990777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:18.990753    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449mzfd" event={"ID":"0e42e185-65bd-44f7-a864-96695117283b","Type":"ContainerDied","Data":"82f6997d5321836c9a3f715c941d845d686ff243bf80466e2923727e74cf4bd4"}
Oct 30 13:06:20.004250 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:20.004208    5093 generic.go:356] "Generic (PLEG): container finished" podID="0e5821d2-c302-4393-8943-72cdff7e41ad" containerID="d1cfcf9fb965d0189081ef8b547be2809229696a1587d12457c8e66a1ba4a031" exitCode=0
Oct 30 13:06:20.004703 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:20.004340    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d8mtd" event={"ID":"0e5821d2-c302-4393-8943-72cdff7e41ad","Type":"ContainerDied","Data":"d1cfcf9fb965d0189081ef8b547be2809229696a1587d12457c8e66a1ba4a031"}
Oct 30 13:06:21.012036 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:21.012001    5093 generic.go:356] "Generic (PLEG): container finished" podID="f3e51708-e19e-424a-8e37-09817a673212" containerID="a392a03f7930554c32e68b59072275ea4c64c31c3eaf98f178ba9887c70c62ac" exitCode=0
Oct 30 13:06:21.012435 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:21.012089    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcn7c" event={"ID":"f3e51708-e19e-424a-8e37-09817a673212","Type":"ContainerDied","Data":"a392a03f7930554c32e68b59072275ea4c64c31c3eaf98f178ba9887c70c62ac"}
Oct 30 13:06:21.014589 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:21.014561    5093 generic.go:356] "Generic (PLEG): container finished" podID="83d191b4-087e-49cb-b73c-2aa61b5a2cbc" containerID="288894b5600d364efeb92d7b74f89a56591089be648772c4ac23609f121b6320" exitCode=0
Oct 30 13:06:21.014685 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:21.014642    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agp879" event={"ID":"83d191b4-087e-49cb-b73c-2aa61b5a2cbc","Type":"ContainerDied","Data":"288894b5600d364efeb92d7b74f89a56591089be648772c4ac23609f121b6320"}
Oct 30 13:06:21.017192 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:21.017117    5093 generic.go:356] "Generic (PLEG): container finished" podID="a2979d0d-8de1-42cf-b174-2dfb76f0bfa6" containerID="ef9814a579f7df4027e6b6cdd7907e726223af9e8ee08c02d63d372f91c636f4" exitCode=0
Oct 30 13:06:21.017271 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:21.017243    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77d5fw" event={"ID":"a2979d0d-8de1-42cf-b174-2dfb76f0bfa6","Type":"ContainerDied","Data":"ef9814a579f7df4027e6b6cdd7907e726223af9e8ee08c02d63d372f91c636f4"}
Oct 30 13:06:21.169334 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:06:21.169285    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:06:22.024462 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:22.024431    5093 generic.go:356] "Generic (PLEG): container finished" podID="6c77c3a6-b076-4cf7-811d-6176b6425e06" containerID="611075f84238e2e1e2625f24be7021fa35765643418f810dc8f2b9d9c5d7471a" exitCode=0
Oct 30 13:06:22.024817 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:22.024519    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqdscd" event={"ID":"6c77c3a6-b076-4cf7-811d-6176b6425e06","Type":"ContainerDied","Data":"611075f84238e2e1e2625f24be7021fa35765643418f810dc8f2b9d9c5d7471a"}
Oct 30 13:06:22.027288 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:22.027265    5093 generic.go:356] "Generic (PLEG): container finished" podID="f3e51708-e19e-424a-8e37-09817a673212" containerID="12b5990b0ee045450e1a3000bba1ede01207c384aa6775049d358269a467f73a" exitCode=0
Oct 30 13:06:22.027381 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:22.027349    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcn7c" event={"ID":"f3e51708-e19e-424a-8e37-09817a673212","Type":"ContainerDied","Data":"12b5990b0ee045450e1a3000bba1ede01207c384aa6775049d358269a467f73a"}
Oct 30 13:06:22.030275 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:22.030253    5093 generic.go:356] "Generic (PLEG): container finished" podID="83d191b4-087e-49cb-b73c-2aa61b5a2cbc" containerID="8c04b91e1905519c63ae2b934a880a2f540b198b50764ed88da087d642f8e0e1" exitCode=0
Oct 30 13:06:22.030365 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:22.030330    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agp879" event={"ID":"83d191b4-087e-49cb-b73c-2aa61b5a2cbc","Type":"ContainerDied","Data":"8c04b91e1905519c63ae2b934a880a2f540b198b50764ed88da087d642f8e0e1"}
Oct 30 13:06:22.032708 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:22.032684    5093 generic.go:356] "Generic (PLEG): container finished" podID="a2979d0d-8de1-42cf-b174-2dfb76f0bfa6" containerID="8e345e744d6bc47c50583232e5f85697b56e064ed11c89654d096dd30c67947f" exitCode=0
Oct 30 13:06:22.032801 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:22.032758    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77d5fw" event={"ID":"a2979d0d-8de1-42cf-b174-2dfb76f0bfa6","Type":"ContainerDied","Data":"8e345e744d6bc47c50583232e5f85697b56e064ed11c89654d096dd30c67947f"}
Oct 30 13:06:22.034996 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:22.034973    5093 generic.go:356] "Generic (PLEG): container finished" podID="0e42e185-65bd-44f7-a864-96695117283b" containerID="d161d3b6fb879ee01cc53efeb4b804be6e505aa2ad90e789032307161a93f099" exitCode=0
Oct 30 13:06:22.035096 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:22.035051    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449mzfd" event={"ID":"0e42e185-65bd-44f7-a864-96695117283b","Type":"ContainerDied","Data":"d161d3b6fb879ee01cc53efeb4b804be6e505aa2ad90e789032307161a93f099"}
Oct 30 13:06:31.123338 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:31.123294    5093 generic.go:356] "Generic (PLEG): container finished" podID="0e5821d2-c302-4393-8943-72cdff7e41ad" containerID="f1081b9d128c6f11a54a5a4808097b80a717ae0f2ae7d98192b2040591ce17dc" exitCode=0
Oct 30 13:06:31.123851 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:31.123398    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d8mtd" event={"ID":"0e5821d2-c302-4393-8943-72cdff7e41ad","Type":"ContainerDied","Data":"f1081b9d128c6f11a54a5a4808097b80a717ae0f2ae7d98192b2040591ce17dc"}
Oct 30 13:06:32.328570 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:32.328533    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d8mtd"
Oct 30 13:06:32.452494 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:32.451625    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e5821d2-c302-4393-8943-72cdff7e41ad-kube-api-access-d8h72\" (UniqueName: \"kubernetes.io/projected/0e5821d2-c302-4393-8943-72cdff7e41ad-kube-api-access-d8h72\") pod \"0e5821d2-c302-4393-8943-72cdff7e41ad\" (UID: \"0e5821d2-c302-4393-8943-72cdff7e41ad\") "
Oct 30 13:06:32.452494 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:32.451735    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e5821d2-c302-4393-8943-72cdff7e41ad-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e5821d2-c302-4393-8943-72cdff7e41ad-bundle\") pod \"0e5821d2-c302-4393-8943-72cdff7e41ad\" (UID: \"0e5821d2-c302-4393-8943-72cdff7e41ad\") "
Oct 30 13:06:32.452494 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:32.451778    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e5821d2-c302-4393-8943-72cdff7e41ad-util\" (UniqueName: \"kubernetes.io/empty-dir/0e5821d2-c302-4393-8943-72cdff7e41ad-util\") pod \"0e5821d2-c302-4393-8943-72cdff7e41ad\" (UID: \"0e5821d2-c302-4393-8943-72cdff7e41ad\") "
Oct 30 13:06:32.453467 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:32.453378    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e5821d2-c302-4393-8943-72cdff7e41ad-bundle" pod "0e5821d2-c302-4393-8943-72cdff7e41ad" (UID: "0e5821d2-c302-4393-8943-72cdff7e41ad"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:32.458424 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:32.458394    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e5821d2-c302-4393-8943-72cdff7e41ad-util" pod "0e5821d2-c302-4393-8943-72cdff7e41ad" (UID: "0e5821d2-c302-4393-8943-72cdff7e41ad"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:32.466617 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:32.466586    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e5821d2-c302-4393-8943-72cdff7e41ad-kube-api-access-d8h72" pod "0e5821d2-c302-4393-8943-72cdff7e41ad" (UID: "0e5821d2-c302-4393-8943-72cdff7e41ad"). InnerVolumeSpecName "kube-api-access-d8h72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:32.552725 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:32.552686    5093 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e5821d2-c302-4393-8943-72cdff7e41ad-bundle\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:06:32.552725 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:32.552726    5093 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e5821d2-c302-4393-8943-72cdff7e41ad-util\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:06:32.552945 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:32.552742    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8h72\" (UniqueName: \"kubernetes.io/projected/0e5821d2-c302-4393-8943-72cdff7e41ad-kube-api-access-d8h72\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:06:33.141368 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:33.141271    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d8mtd" event={"ID":"0e5821d2-c302-4393-8943-72cdff7e41ad","Type":"ContainerDied","Data":"e5abb99dd96e1ad2a9017d3f3be0025078e8209f2aff2ac84fd847324c73dc44"}
Oct 30 13:06:33.141368 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:33.141339    5093 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5abb99dd96e1ad2a9017d3f3be0025078e8209f2aff2ac84fd847324c73dc44"
Oct 30 13:06:33.142642 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:33.141962    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d8mtd"
Oct 30 13:06:35.155767 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:35.155733    5093 generic.go:356] "Generic (PLEG): container finished" podID="f3e51708-e19e-424a-8e37-09817a673212" containerID="b8174ca65c60d1e3404cb9fda15814aabeb291aae63e552aab83f853cea91f2a" exitCode=0
Oct 30 13:06:35.156211 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:35.155819    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcn7c" event={"ID":"f3e51708-e19e-424a-8e37-09817a673212","Type":"ContainerDied","Data":"b8174ca65c60d1e3404cb9fda15814aabeb291aae63e552aab83f853cea91f2a"}
Oct 30 13:06:35.157893 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:35.157857    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agp879" event={"ID":"83d191b4-087e-49cb-b73c-2aa61b5a2cbc","Type":"ContainerStarted","Data":"4c0aabcf7b80d45578f1872faec0545561a67520e690176c673a5df9b5c4f768"}
Oct 30 13:06:35.159762 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:35.159740    5093 generic.go:356] "Generic (PLEG): container finished" podID="a2979d0d-8de1-42cf-b174-2dfb76f0bfa6" containerID="1ecd2b2f0d6cb86e6ecef25e58afe35b6e2ba2feaae20b2448e3032832a3f0bb" exitCode=0
Oct 30 13:06:35.159857 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:35.159807    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77d5fw" event={"ID":"a2979d0d-8de1-42cf-b174-2dfb76f0bfa6","Type":"ContainerDied","Data":"1ecd2b2f0d6cb86e6ecef25e58afe35b6e2ba2feaae20b2448e3032832a3f0bb"}
Oct 30 13:06:35.161759 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:35.161740    5093 generic.go:356] "Generic (PLEG): container finished" podID="0e42e185-65bd-44f7-a864-96695117283b" containerID="49324418eb2ffc65a27469cf56f3aa4d2c37ad2164d565305f242269426444b3" exitCode=0
Oct 30 13:06:35.161846 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:35.161816    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449mzfd" event={"ID":"0e42e185-65bd-44f7-a864-96695117283b","Type":"ContainerDied","Data":"49324418eb2ffc65a27469cf56f3aa4d2c37ad2164d565305f242269426444b3"}
Oct 30 13:06:35.163779 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:35.163761    5093 generic.go:356] "Generic (PLEG): container finished" podID="6c77c3a6-b076-4cf7-811d-6176b6425e06" containerID="3e4c8710d9eb54076772893cfbd5b08563fac2e30cd1d0f9772b070991984c56" exitCode=0
Oct 30 13:06:35.163874 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:35.163787    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqdscd" event={"ID":"6c77c3a6-b076-4cf7-811d-6176b6425e06","Type":"ContainerDied","Data":"3e4c8710d9eb54076772893cfbd5b08563fac2e30cd1d0f9772b070991984c56"}
Oct 30 13:06:35.286439 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:35.286331    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agp879" podStartSLOduration=2.843087811 podStartE2EDuration="28.286314868s" totalImagesPullingTime="25.443227057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:35.286066987 +0000 UTC m=+214.433334239" watchObservedRunningTime="2026-10-30 13:06:35.286314868 +0000 UTC m=+214.433582107"
Oct 30 13:06:36.169959 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:36.169864    5093 generic.go:356] "Generic (PLEG): container finished" podID="83d191b4-087e-49cb-b73c-2aa61b5a2cbc" containerID="4c0aabcf7b80d45578f1872faec0545561a67520e690176c673a5df9b5c4f768" exitCode=0
Oct 30 13:06:36.169959 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:36.169949    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agp879" event={"ID":"83d191b4-087e-49cb-b73c-2aa61b5a2cbc","Type":"ContainerDied","Data":"4c0aabcf7b80d45578f1872faec0545561a67520e690176c673a5df9b5c4f768"}
Oct 30 13:06:36.323720 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:36.323690    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449mzfd"
Oct 30 13:06:36.349437 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:36.349412    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcn7c"
Oct 30 13:06:36.352289 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:36.352268    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77d5fw"
Oct 30 13:06:36.384706 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:36.384669    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqdscd"
Oct 30 13:06:36.488596 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:36.488557    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2979d0d-8de1-42cf-b174-2dfb76f0bfa6-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2979d0d-8de1-42cf-b174-2dfb76f0bfa6-bundle\") pod \"a2979d0d-8de1-42cf-b174-2dfb76f0bfa6\" (UID: \"a2979d0d-8de1-42cf-b174-2dfb76f0bfa6\") "
Oct 30 13:06:36.488596 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:36.488601    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c77c3a6-b076-4cf7-811d-6176b6425e06-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6c77c3a6-b076-4cf7-811d-6176b6425e06-bundle\") pod \"6c77c3a6-b076-4cf7-811d-6176b6425e06\" (UID: \"6c77c3a6-b076-4cf7-811d-6176b6425e06\") "
Oct 30 13:06:36.488851 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:36.488633    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2979d0d-8de1-42cf-b174-2dfb76f0bfa6-util\" (UniqueName: \"kubernetes.io/empty-dir/a2979d0d-8de1-42cf-b174-2dfb76f0bfa6-util\") pod \"a2979d0d-8de1-42cf-b174-2dfb76f0bfa6\" (UID: \"a2979d0d-8de1-42cf-b174-2dfb76f0bfa6\") "
Oct 30 13:06:36.488851 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:36.488673    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3e51708-e19e-424a-8e37-09817a673212-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f3e51708-e19e-424a-8e37-09817a673212-bundle\") pod \"f3e51708-e19e-424a-8e37-09817a673212\" (UID: \"f3e51708-e19e-424a-8e37-09817a673212\") "
Oct 30 13:06:36.488851 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:36.488702    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2979d0d-8de1-42cf-b174-2dfb76f0bfa6-kube-api-access-jsbct\" (UniqueName: \"kubernetes.io/projected/a2979d0d-8de1-42cf-b174-2dfb76f0bfa6-kube-api-access-jsbct\") pod \"a2979d0d-8de1-42cf-b174-2dfb76f0bfa6\" (UID: \"a2979d0d-8de1-42cf-b174-2dfb76f0bfa6\") "
Oct 30 13:06:36.488851 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:36.488732    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e42e185-65bd-44f7-a864-96695117283b-util\" (UniqueName: \"kubernetes.io/empty-dir/0e42e185-65bd-44f7-a864-96695117283b-util\") pod \"0e42e185-65bd-44f7-a864-96695117283b\" (UID: \"0e42e185-65bd-44f7-a864-96695117283b\") "
Oct 30 13:06:36.488851 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:36.488757    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e42e185-65bd-44f7-a864-96695117283b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e42e185-65bd-44f7-a864-96695117283b-bundle\") pod \"0e42e185-65bd-44f7-a864-96695117283b\" (UID: \"0e42e185-65bd-44f7-a864-96695117283b\") "
Oct 30 13:06:36.488851 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:36.488846    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e42e185-65bd-44f7-a864-96695117283b-kube-api-access-mzlzf\" (UniqueName: \"kubernetes.io/projected/0e42e185-65bd-44f7-a864-96695117283b-kube-api-access-mzlzf\") pod \"0e42e185-65bd-44f7-a864-96695117283b\" (UID: \"0e42e185-65bd-44f7-a864-96695117283b\") "
Oct 30 13:06:36.489158 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:36.488899    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3e51708-e19e-424a-8e37-09817a673212-kube-api-access-cv4qr\" (UniqueName: \"kubernetes.io/projected/f3e51708-e19e-424a-8e37-09817a673212-kube-api-access-cv4qr\") pod \"f3e51708-e19e-424a-8e37-09817a673212\" (UID: \"f3e51708-e19e-424a-8e37-09817a673212\") "
Oct 30 13:06:36.489158 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:36.488929    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c77c3a6-b076-4cf7-811d-6176b6425e06-util\" (UniqueName: \"kubernetes.io/empty-dir/6c77c3a6-b076-4cf7-811d-6176b6425e06-util\") pod \"6c77c3a6-b076-4cf7-811d-6176b6425e06\" (UID: \"6c77c3a6-b076-4cf7-811d-6176b6425e06\") "
Oct 30 13:06:36.489158 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:36.488965    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3e51708-e19e-424a-8e37-09817a673212-util\" (UniqueName: \"kubernetes.io/empty-dir/f3e51708-e19e-424a-8e37-09817a673212-util\") pod \"f3e51708-e19e-424a-8e37-09817a673212\" (UID: \"f3e51708-e19e-424a-8e37-09817a673212\") "
Oct 30 13:06:36.489158 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:36.489000    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c77c3a6-b076-4cf7-811d-6176b6425e06-kube-api-access-dqxkd\" (UniqueName: \"kubernetes.io/projected/6c77c3a6-b076-4cf7-811d-6176b6425e06-kube-api-access-dqxkd\") pod \"6c77c3a6-b076-4cf7-811d-6176b6425e06\" (UID: \"6c77c3a6-b076-4cf7-811d-6176b6425e06\") "
Oct 30 13:06:36.489361 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:36.489216    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2979d0d-8de1-42cf-b174-2dfb76f0bfa6-bundle" pod "a2979d0d-8de1-42cf-b174-2dfb76f0bfa6" (UID: "a2979d0d-8de1-42cf-b174-2dfb76f0bfa6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:36.489361 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:36.489285    5093 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2979d0d-8de1-42cf-b174-2dfb76f0bfa6-bundle\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:06:36.489817 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:36.489658    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3e51708-e19e-424a-8e37-09817a673212-bundle" pod "f3e51708-e19e-424a-8e37-09817a673212" (UID: "f3e51708-e19e-424a-8e37-09817a673212"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:36.489925 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:36.489816    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c77c3a6-b076-4cf7-811d-6176b6425e06-bundle" pod "6c77c3a6-b076-4cf7-811d-6176b6425e06" (UID: "6c77c3a6-b076-4cf7-811d-6176b6425e06"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:36.490121 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:36.490102    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e42e185-65bd-44f7-a864-96695117283b-bundle" pod "0e42e185-65bd-44f7-a864-96695117283b" (UID: "0e42e185-65bd-44f7-a864-96695117283b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:36.492455 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:36.492428    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c77c3a6-b076-4cf7-811d-6176b6425e06-kube-api-access-dqxkd" pod "6c77c3a6-b076-4cf7-811d-6176b6425e06" (UID: "6c77c3a6-b076-4cf7-811d-6176b6425e06"). InnerVolumeSpecName "kube-api-access-dqxkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:36.492689 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:36.492670    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3e51708-e19e-424a-8e37-09817a673212-kube-api-access-cv4qr" pod "f3e51708-e19e-424a-8e37-09817a673212" (UID: "f3e51708-e19e-424a-8e37-09817a673212"). InnerVolumeSpecName "kube-api-access-cv4qr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:36.492935 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:36.492913    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e42e185-65bd-44f7-a864-96695117283b-kube-api-access-mzlzf" pod "0e42e185-65bd-44f7-a864-96695117283b" (UID: "0e42e185-65bd-44f7-a864-96695117283b"). InnerVolumeSpecName "kube-api-access-mzlzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:36.494985 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:36.494958    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2979d0d-8de1-42cf-b174-2dfb76f0bfa6-kube-api-access-jsbct" pod "a2979d0d-8de1-42cf-b174-2dfb76f0bfa6" (UID: "a2979d0d-8de1-42cf-b174-2dfb76f0bfa6"). InnerVolumeSpecName "kube-api-access-jsbct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:36.495245 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:36.495220    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2979d0d-8de1-42cf-b174-2dfb76f0bfa6-util" pod "a2979d0d-8de1-42cf-b174-2dfb76f0bfa6" (UID: "a2979d0d-8de1-42cf-b174-2dfb76f0bfa6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:36.496262 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:36.496240    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c77c3a6-b076-4cf7-811d-6176b6425e06-util" pod "6c77c3a6-b076-4cf7-811d-6176b6425e06" (UID: "6c77c3a6-b076-4cf7-811d-6176b6425e06"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:36.496356 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:36.496343    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3e51708-e19e-424a-8e37-09817a673212-util" pod "f3e51708-e19e-424a-8e37-09817a673212" (UID: "f3e51708-e19e-424a-8e37-09817a673212"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:36.496875 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:36.496848    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e42e185-65bd-44f7-a864-96695117283b-util" pod "0e42e185-65bd-44f7-a864-96695117283b" (UID: "0e42e185-65bd-44f7-a864-96695117283b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:36.590277 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:36.590238    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv4qr\" (UniqueName: \"kubernetes.io/projected/f3e51708-e19e-424a-8e37-09817a673212-kube-api-access-cv4qr\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:06:36.590277 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:36.590266    5093 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6c77c3a6-b076-4cf7-811d-6176b6425e06-util\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:06:36.590277 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:36.590278    5093 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f3e51708-e19e-424a-8e37-09817a673212-util\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:06:36.590277 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:36.590287    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqxkd\" (UniqueName: \"kubernetes.io/projected/6c77c3a6-b076-4cf7-811d-6176b6425e06-kube-api-access-dqxkd\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:06:36.590536 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:36.590295    5093 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6c77c3a6-b076-4cf7-811d-6176b6425e06-bundle\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:06:36.590536 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:36.590304    5093 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2979d0d-8de1-42cf-b174-2dfb76f0bfa6-util\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:06:36.590536 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:36.590311    5093 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f3e51708-e19e-424a-8e37-09817a673212-bundle\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:06:36.590536 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:36.590318    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsbct\" (UniqueName: \"kubernetes.io/projected/a2979d0d-8de1-42cf-b174-2dfb76f0bfa6-kube-api-access-jsbct\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:06:36.590536 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:36.590326    5093 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e42e185-65bd-44f7-a864-96695117283b-util\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:06:36.590536 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:36.590333    5093 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e42e185-65bd-44f7-a864-96695117283b-bundle\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:06:36.590536 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:36.590342    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzlzf\" (UniqueName: \"kubernetes.io/projected/0e42e185-65bd-44f7-a864-96695117283b-kube-api-access-mzlzf\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:06:37.177212 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.177167    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqdscd" event={"ID":"6c77c3a6-b076-4cf7-811d-6176b6425e06","Type":"ContainerDied","Data":"cb28d7a5fe157ab8180d74636731b4633dd3d3e1fb8ebbbb3901e793c89ef0f6"}
Oct 30 13:06:37.177212 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.177191    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqdscd"
Oct 30 13:06:37.177643 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.177227    5093 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb28d7a5fe157ab8180d74636731b4633dd3d3e1fb8ebbbb3901e793c89ef0f6"
Oct 30 13:06:37.180302 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.179757    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcn7c"
Oct 30 13:06:37.180302 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.179788    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcn7c" event={"ID":"f3e51708-e19e-424a-8e37-09817a673212","Type":"ContainerDied","Data":"7ec81e1879f2c9d3a5bcb4b2b9e15616ca26ffe60e4838466832bed890a9c00c"}
Oct 30 13:06:37.180302 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.179831    5093 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ec81e1879f2c9d3a5bcb4b2b9e15616ca26ffe60e4838466832bed890a9c00c"
Oct 30 13:06:37.182149 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.182030    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77d5fw" event={"ID":"a2979d0d-8de1-42cf-b174-2dfb76f0bfa6","Type":"ContainerDied","Data":"f0b640397adc10de083fed3837bc9787c453fcae00c15e2aab0f431afa3860f7"}
Oct 30 13:06:37.182149 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.182067    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77d5fw"
Oct 30 13:06:37.182149 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.182083    5093 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0b640397adc10de083fed3837bc9787c453fcae00c15e2aab0f431afa3860f7"
Oct 30 13:06:37.184620 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.184582    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449mzfd"
Oct 30 13:06:37.184969 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.184949    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449mzfd" event={"ID":"0e42e185-65bd-44f7-a864-96695117283b","Type":"ContainerDied","Data":"e78d056dc516129670d51c4d307cc8e8ac80ae0623d521f616a7f818cd1c45a0"}
Oct 30 13:06:37.185091 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.184986    5093 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e78d056dc516129670d51c4d307cc8e8ac80ae0623d521f616a7f818cd1c45a0"
Oct 30 13:06:37.265474 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.265390    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-rd5mx"]
Oct 30 13:06:37.265711 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.265698    5093 cpu_manager.go:414] "containerMap: removing container" podUID="0e42e185-65bd-44f7-a864-96695117283b" containerName="util"
Oct 30 13:06:37.265777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.265711    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e42e185-65bd-44f7-a864-96695117283b" containerName="util"
Oct 30 13:06:37.265777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.265723    5093 cpu_manager.go:414] "containerMap: removing container" podUID="0e5821d2-c302-4393-8943-72cdff7e41ad" containerName="pull"
Oct 30 13:06:37.265777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.265728    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e5821d2-c302-4393-8943-72cdff7e41ad" containerName="pull"
Oct 30 13:06:37.265777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.265735    5093 cpu_manager.go:414] "containerMap: removing container" podUID="f3e51708-e19e-424a-8e37-09817a673212" containerName="pull"
Oct 30 13:06:37.265777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.265740    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3e51708-e19e-424a-8e37-09817a673212" containerName="pull"
Oct 30 13:06:37.265777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.265749    5093 cpu_manager.go:414] "containerMap: removing container" podUID="a2979d0d-8de1-42cf-b174-2dfb76f0bfa6" containerName="util"
Oct 30 13:06:37.265777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.265754    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2979d0d-8de1-42cf-b174-2dfb76f0bfa6" containerName="util"
Oct 30 13:06:37.265777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.265761    5093 cpu_manager.go:414] "containerMap: removing container" podUID="0e5821d2-c302-4393-8943-72cdff7e41ad" containerName="extract"
Oct 30 13:06:37.265777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.265766    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e5821d2-c302-4393-8943-72cdff7e41ad" containerName="extract"
Oct 30 13:06:37.265777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.265773    5093 cpu_manager.go:414] "containerMap: removing container" podUID="6c77c3a6-b076-4cf7-811d-6176b6425e06" containerName="pull"
Oct 30 13:06:37.265777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.265779    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c77c3a6-b076-4cf7-811d-6176b6425e06" containerName="pull"
Oct 30 13:06:37.265777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.265786    5093 cpu_manager.go:414] "containerMap: removing container" podUID="a2979d0d-8de1-42cf-b174-2dfb76f0bfa6" containerName="extract"
Oct 30 13:06:37.266376 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.265791    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2979d0d-8de1-42cf-b174-2dfb76f0bfa6" containerName="extract"
Oct 30 13:06:37.266376 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.265800    5093 cpu_manager.go:414] "containerMap: removing container" podUID="f3e51708-e19e-424a-8e37-09817a673212" containerName="extract"
Oct 30 13:06:37.266376 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.265804    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3e51708-e19e-424a-8e37-09817a673212" containerName="extract"
Oct 30 13:06:37.266376 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.265811    5093 cpu_manager.go:414] "containerMap: removing container" podUID="0e42e185-65bd-44f7-a864-96695117283b" containerName="extract"
Oct 30 13:06:37.266376 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.265816    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e42e185-65bd-44f7-a864-96695117283b" containerName="extract"
Oct 30 13:06:37.266376 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.265828    5093 cpu_manager.go:414] "containerMap: removing container" podUID="6c77c3a6-b076-4cf7-811d-6176b6425e06" containerName="util"
Oct 30 13:06:37.266376 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.265832    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c77c3a6-b076-4cf7-811d-6176b6425e06" containerName="util"
Oct 30 13:06:37.266376 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.265838    5093 cpu_manager.go:414] "containerMap: removing container" podUID="0e5821d2-c302-4393-8943-72cdff7e41ad" containerName="util"
Oct 30 13:06:37.266376 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.265842    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e5821d2-c302-4393-8943-72cdff7e41ad" containerName="util"
Oct 30 13:06:37.266376 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.265848    5093 cpu_manager.go:414] "containerMap: removing container" podUID="a2979d0d-8de1-42cf-b174-2dfb76f0bfa6" containerName="pull"
Oct 30 13:06:37.266376 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.265852    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2979d0d-8de1-42cf-b174-2dfb76f0bfa6" containerName="pull"
Oct 30 13:06:37.266376 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.265860    5093 cpu_manager.go:414] "containerMap: removing container" podUID="6c77c3a6-b076-4cf7-811d-6176b6425e06" containerName="extract"
Oct 30 13:06:37.266376 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.265864    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c77c3a6-b076-4cf7-811d-6176b6425e06" containerName="extract"
Oct 30 13:06:37.266376 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.265872    5093 cpu_manager.go:414] "containerMap: removing container" podUID="f3e51708-e19e-424a-8e37-09817a673212" containerName="util"
Oct 30 13:06:37.266376 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.265876    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3e51708-e19e-424a-8e37-09817a673212" containerName="util"
Oct 30 13:06:37.266376 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.265887    5093 cpu_manager.go:414] "containerMap: removing container" podUID="0e42e185-65bd-44f7-a864-96695117283b" containerName="pull"
Oct 30 13:06:37.266376 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.265891    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e42e185-65bd-44f7-a864-96695117283b" containerName="pull"
Oct 30 13:06:37.266376 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.265952    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c77c3a6-b076-4cf7-811d-6176b6425e06" containerName="extract"
Oct 30 13:06:37.266376 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.265961    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e5821d2-c302-4393-8943-72cdff7e41ad" containerName="extract"
Oct 30 13:06:37.266376 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.265968    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2979d0d-8de1-42cf-b174-2dfb76f0bfa6" containerName="extract"
Oct 30 13:06:37.266376 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.265973    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3e51708-e19e-424a-8e37-09817a673212" containerName="extract"
Oct 30 13:06:37.266376 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.265979    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e42e185-65bd-44f7-a864-96695117283b" containerName="extract"
Oct 30 13:06:37.270382 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.270360    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-rd5mx"
Oct 30 13:06:37.298109 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.298078    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 13:06:37.298238 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.298223    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 13:06:37.298293 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.298274    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-gfqwn\""
Oct 30 13:06:37.305171 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.305148    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 13:06:37.311287 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.311269    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agp879"
Oct 30 13:06:37.311605 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.311588    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 13:06:37.311644 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.311589    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 13:06:37.332683 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.332653    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-rd5mx"]
Oct 30 13:06:37.396933 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.396900    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83d191b4-087e-49cb-b73c-2aa61b5a2cbc-kube-api-access-d4bwq\" (UniqueName: \"kubernetes.io/projected/83d191b4-087e-49cb-b73c-2aa61b5a2cbc-kube-api-access-d4bwq\") pod \"83d191b4-087e-49cb-b73c-2aa61b5a2cbc\" (UID: \"83d191b4-087e-49cb-b73c-2aa61b5a2cbc\") "
Oct 30 13:06:37.397105 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.396956    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83d191b4-087e-49cb-b73c-2aa61b5a2cbc-bundle\" (UniqueName: \"kubernetes.io/empty-dir/83d191b4-087e-49cb-b73c-2aa61b5a2cbc-bundle\") pod \"83d191b4-087e-49cb-b73c-2aa61b5a2cbc\" (UID: \"83d191b4-087e-49cb-b73c-2aa61b5a2cbc\") "
Oct 30 13:06:37.397105 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.397016    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83d191b4-087e-49cb-b73c-2aa61b5a2cbc-util\" (UniqueName: \"kubernetes.io/empty-dir/83d191b4-087e-49cb-b73c-2aa61b5a2cbc-util\") pod \"83d191b4-087e-49cb-b73c-2aa61b5a2cbc\" (UID: \"83d191b4-087e-49cb-b73c-2aa61b5a2cbc\") "
Oct 30 13:06:37.397214 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.397168    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/37f62437-e885-4923-9c4b-06fd2ff0af63-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-rd5mx\" (UID: \"37f62437-e885-4923-9c4b-06fd2ff0af63\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-rd5mx"
Oct 30 13:06:37.397214 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.397207    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm5rp\" (UniqueName: \"kubernetes.io/projected/37f62437-e885-4923-9c4b-06fd2ff0af63-kube-api-access-zm5rp\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-rd5mx\" (UID: \"37f62437-e885-4923-9c4b-06fd2ff0af63\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-rd5mx"
Oct 30 13:06:37.397335 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.397243    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/37f62437-e885-4923-9c4b-06fd2ff0af63-manifests\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-rd5mx\" (UID: \"37f62437-e885-4923-9c4b-06fd2ff0af63\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-rd5mx"
Oct 30 13:06:37.397448 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.397427    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/37f62437-e885-4923-9c4b-06fd2ff0af63-metrics-cert\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-rd5mx\" (UID: \"37f62437-e885-4923-9c4b-06fd2ff0af63\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-rd5mx"
Oct 30 13:06:37.397541 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.397517    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/37f62437-e885-4923-9c4b-06fd2ff0af63-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-rd5mx\" (UID: \"37f62437-e885-4923-9c4b-06fd2ff0af63\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-rd5mx"
Oct 30 13:06:37.397584 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.397569    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37f62437-e885-4923-9c4b-06fd2ff0af63-tmp\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-rd5mx\" (UID: \"37f62437-e885-4923-9c4b-06fd2ff0af63\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-rd5mx"
Oct 30 13:06:37.397767 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.397749    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83d191b4-087e-49cb-b73c-2aa61b5a2cbc-bundle" pod "83d191b4-087e-49cb-b73c-2aa61b5a2cbc" (UID: "83d191b4-087e-49cb-b73c-2aa61b5a2cbc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:37.399084 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.399062    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83d191b4-087e-49cb-b73c-2aa61b5a2cbc-kube-api-access-d4bwq" pod "83d191b4-087e-49cb-b73c-2aa61b5a2cbc" (UID: "83d191b4-087e-49cb-b73c-2aa61b5a2cbc"). InnerVolumeSpecName "kube-api-access-d4bwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:37.402736 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.402712    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83d191b4-087e-49cb-b73c-2aa61b5a2cbc-util" pod "83d191b4-087e-49cb-b73c-2aa61b5a2cbc" (UID: "83d191b4-087e-49cb-b73c-2aa61b5a2cbc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:37.498011 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.497968    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/37f62437-e885-4923-9c4b-06fd2ff0af63-metrics-cert\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-rd5mx\" (UID: \"37f62437-e885-4923-9c4b-06fd2ff0af63\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-rd5mx"
Oct 30 13:06:37.498011 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.498012    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/37f62437-e885-4923-9c4b-06fd2ff0af63-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-rd5mx\" (UID: \"37f62437-e885-4923-9c4b-06fd2ff0af63\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-rd5mx"
Oct 30 13:06:37.498294 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.498033    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37f62437-e885-4923-9c4b-06fd2ff0af63-tmp\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-rd5mx\" (UID: \"37f62437-e885-4923-9c4b-06fd2ff0af63\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-rd5mx"
Oct 30 13:06:37.498294 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.498069    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/37f62437-e885-4923-9c4b-06fd2ff0af63-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-rd5mx\" (UID: \"37f62437-e885-4923-9c4b-06fd2ff0af63\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-rd5mx"
Oct 30 13:06:37.498294 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.498089    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm5rp\" (UniqueName: \"kubernetes.io/projected/37f62437-e885-4923-9c4b-06fd2ff0af63-kube-api-access-zm5rp\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-rd5mx\" (UID: \"37f62437-e885-4923-9c4b-06fd2ff0af63\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-rd5mx"
Oct 30 13:06:37.498294 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.498123    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/37f62437-e885-4923-9c4b-06fd2ff0af63-manifests\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-rd5mx\" (UID: \"37f62437-e885-4923-9c4b-06fd2ff0af63\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-rd5mx"
Oct 30 13:06:37.498294 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.498238    5093 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83d191b4-087e-49cb-b73c-2aa61b5a2cbc-bundle\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:06:37.498294 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.498255    5093 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83d191b4-087e-49cb-b73c-2aa61b5a2cbc-util\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:06:37.498294 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.498268    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4bwq\" (UniqueName: \"kubernetes.io/projected/83d191b4-087e-49cb-b73c-2aa61b5a2cbc-kube-api-access-d4bwq\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:06:37.498651 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.498595    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/37f62437-e885-4923-9c4b-06fd2ff0af63-manifests\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-rd5mx\" (UID: \"37f62437-e885-4923-9c4b-06fd2ff0af63\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-rd5mx"
Oct 30 13:06:37.500495 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.500467    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37f62437-e885-4923-9c4b-06fd2ff0af63-tmp\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-rd5mx\" (UID: \"37f62437-e885-4923-9c4b-06fd2ff0af63\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-rd5mx"
Oct 30 13:06:37.500852 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.500821    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/37f62437-e885-4923-9c4b-06fd2ff0af63-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-rd5mx\" (UID: \"37f62437-e885-4923-9c4b-06fd2ff0af63\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-rd5mx"
Oct 30 13:06:37.500968 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.500947    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/37f62437-e885-4923-9c4b-06fd2ff0af63-metrics-cert\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-rd5mx\" (UID: \"37f62437-e885-4923-9c4b-06fd2ff0af63\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-rd5mx"
Oct 30 13:06:37.501050 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.501030    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/37f62437-e885-4923-9c4b-06fd2ff0af63-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-rd5mx\" (UID: \"37f62437-e885-4923-9c4b-06fd2ff0af63\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-rd5mx"
Oct 30 13:06:37.522145 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.522063    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm5rp\" (UniqueName: \"kubernetes.io/projected/37f62437-e885-4923-9c4b-06fd2ff0af63-kube-api-access-zm5rp\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-rd5mx\" (UID: \"37f62437-e885-4923-9c4b-06fd2ff0af63\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-rd5mx"
Oct 30 13:06:37.585792 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.585762    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-rd5mx"
Oct 30 13:06:37.917195 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:37.917166    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-rd5mx"]
Oct 30 13:06:37.919988 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:06:37.919953    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37f62437_e885_4923_9c4b_06fd2ff0af63.slice/crio-515b3a95a8c6018dcaec2747f209bafc8d1299e8aea8d9101c44776ce087920b WatchSource:0}: Error finding container 515b3a95a8c6018dcaec2747f209bafc8d1299e8aea8d9101c44776ce087920b: Status 404 returned error can't find the container with id 515b3a95a8c6018dcaec2747f209bafc8d1299e8aea8d9101c44776ce087920b
Oct 30 13:06:38.190611 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:38.190575    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-rd5mx" event={"ID":"37f62437-e885-4923-9c4b-06fd2ff0af63","Type":"ContainerStarted","Data":"515b3a95a8c6018dcaec2747f209bafc8d1299e8aea8d9101c44776ce087920b"}
Oct 30 13:06:38.192584 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:38.192561    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agp879" event={"ID":"83d191b4-087e-49cb-b73c-2aa61b5a2cbc","Type":"ContainerDied","Data":"6398625cabff59ebd6349a5a5fedca2915f74ab833daaacc6a918a243d52ed99"}
Oct 30 13:06:38.192700 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:38.192598    5093 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6398625cabff59ebd6349a5a5fedca2915f74ab833daaacc6a918a243d52ed99"
Oct 30 13:06:38.192700 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:38.192603    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agp879"
Oct 30 13:06:41.209037 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:41.208996    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-rd5mx" event={"ID":"37f62437-e885-4923-9c4b-06fd2ff0af63","Type":"ContainerStarted","Data":"d5255156b8c88ba49a81ee0abd69d77ecb794e0e14f69c452287298469fe0823"}
Oct 30 13:06:42.215861 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:42.215826    5093 generic.go:356] "Generic (PLEG): container finished" podID="37f62437-e885-4923-9c4b-06fd2ff0af63" containerID="d5255156b8c88ba49a81ee0abd69d77ecb794e0e14f69c452287298469fe0823" exitCode=0
Oct 30 13:06:42.216395 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:42.215957    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-rd5mx" event={"ID":"37f62437-e885-4923-9c4b-06fd2ff0af63","Type":"ContainerDied","Data":"d5255156b8c88ba49a81ee0abd69d77ecb794e0e14f69c452287298469fe0823"}
Oct 30 13:06:43.223107 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:43.223066    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-rd5mx" event={"ID":"37f62437-e885-4923-9c4b-06fd2ff0af63","Type":"ContainerStarted","Data":"fbc502f5e02d0881b6db0bfb3161bc806197780f4b6544a79b82178a0d420847"}
Oct 30 13:06:43.223606 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:43.223208    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-rd5mx"
Oct 30 13:06:43.313667 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:43.313610    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-rd5mx" podStartSLOduration=2.176298369 podStartE2EDuration="6.313595577s" totalImagesPullingTime="3.6601892s" totalInitContainerRuntime="477.108008ms" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:43.299247614 +0000 UTC m=+222.446514854" watchObservedRunningTime="2026-10-30 13:06:43.313595577 +0000 UTC m=+222.460862878"
Oct 30 13:06:44.385198 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:44.385163    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xwmh9"]
Oct 30 13:06:44.385578 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:44.385480    5093 cpu_manager.go:414] "containerMap: removing container" podUID="83d191b4-087e-49cb-b73c-2aa61b5a2cbc" containerName="pull"
Oct 30 13:06:44.385578 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:44.385490    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83d191b4-087e-49cb-b73c-2aa61b5a2cbc" containerName="pull"
Oct 30 13:06:44.385578 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:44.385511    5093 cpu_manager.go:414] "containerMap: removing container" podUID="83d191b4-087e-49cb-b73c-2aa61b5a2cbc" containerName="extract"
Oct 30 13:06:44.385578 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:44.385517    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83d191b4-087e-49cb-b73c-2aa61b5a2cbc" containerName="extract"
Oct 30 13:06:44.385578 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:44.385525    5093 cpu_manager.go:414] "containerMap: removing container" podUID="83d191b4-087e-49cb-b73c-2aa61b5a2cbc" containerName="util"
Oct 30 13:06:44.385578 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:44.385530    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83d191b4-087e-49cb-b73c-2aa61b5a2cbc" containerName="util"
Oct 30 13:06:44.385775 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:44.385588    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="83d191b4-087e-49cb-b73c-2aa61b5a2cbc" containerName="extract"
Oct 30 13:06:44.387523 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:44.387508    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xwmh9"
Oct 30 13:06:44.401149 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:44.401102    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xwmh9"]
Oct 30 13:06:44.412640 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:44.412604    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Oct 30 13:06:44.412640 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:44.412604    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Oct 30 13:06:44.412807 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:44.412646    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-jtmmb\""
Oct 30 13:06:44.459938 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:44.459899    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzb2n\" (UniqueName: \"kubernetes.io/projected/a320be5c-0f19-4dac-9826-25d53c4a9bfc-kube-api-access-wzb2n\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-xwmh9\" (UID: \"a320be5c-0f19-4dac-9826-25d53c4a9bfc\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xwmh9"
Oct 30 13:06:44.460109 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:44.460012    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a320be5c-0f19-4dac-9826-25d53c4a9bfc-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-xwmh9\" (UID: \"a320be5c-0f19-4dac-9826-25d53c4a9bfc\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xwmh9"
Oct 30 13:06:44.560815 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:44.560780    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a320be5c-0f19-4dac-9826-25d53c4a9bfc-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-xwmh9\" (UID: \"a320be5c-0f19-4dac-9826-25d53c4a9bfc\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xwmh9"
Oct 30 13:06:44.560958 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:44.560832    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzb2n\" (UniqueName: \"kubernetes.io/projected/a320be5c-0f19-4dac-9826-25d53c4a9bfc-kube-api-access-wzb2n\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-xwmh9\" (UID: \"a320be5c-0f19-4dac-9826-25d53c4a9bfc\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xwmh9"
Oct 30 13:06:44.561189 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:44.561167    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a320be5c-0f19-4dac-9826-25d53c4a9bfc-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-xwmh9\" (UID: \"a320be5c-0f19-4dac-9826-25d53c4a9bfc\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xwmh9"
Oct 30 13:06:44.584912 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:44.584855    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzb2n\" (UniqueName: \"kubernetes.io/projected/a320be5c-0f19-4dac-9826-25d53c4a9bfc-kube-api-access-wzb2n\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-xwmh9\" (UID: \"a320be5c-0f19-4dac-9826-25d53c4a9bfc\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xwmh9"
Oct 30 13:06:44.698350 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:44.698302    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xwmh9"
Oct 30 13:06:45.028302 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:45.028278    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xwmh9"]
Oct 30 13:06:45.030303 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:06:45.030273    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda320be5c_0f19_4dac_9826_25d53c4a9bfc.slice/crio-2ade9ea0e3fee0905c3b7bfb8705152b52a63c4ebec05631619c345ee639a15f WatchSource:0}: Error finding container 2ade9ea0e3fee0905c3b7bfb8705152b52a63c4ebec05631619c345ee639a15f: Status 404 returned error can't find the container with id 2ade9ea0e3fee0905c3b7bfb8705152b52a63c4ebec05631619c345ee639a15f
Oct 30 13:06:45.233604 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:45.233570    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xwmh9" event={"ID":"a320be5c-0f19-4dac-9826-25d53c4a9bfc","Type":"ContainerStarted","Data":"2ade9ea0e3fee0905c3b7bfb8705152b52a63c4ebec05631619c345ee639a15f"}
Oct 30 13:06:47.248982 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:47.248944    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xwmh9" event={"ID":"a320be5c-0f19-4dac-9826-25d53c4a9bfc","Type":"ContainerStarted","Data":"d12f9c9a90170dda6236d5325057e72360ad90aee1725e5112ea784a53b8bcbc"}
Oct 30 13:06:47.355232 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:47.355175    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xwmh9" podStartSLOduration=1.924171722 podStartE2EDuration="3.355160913s" totalImagesPullingTime="1.430989191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:47.354675631 +0000 UTC m=+226.501942872" watchObservedRunningTime="2026-10-30 13:06:47.355160913 +0000 UTC m=+226.502428150"
Oct 30 13:06:49.731144 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:49.731097    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-8269v"]
Oct 30 13:06:49.737542 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:49.737521    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8269v"
Oct 30 13:06:49.753847 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:49.753818    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-8269v"]
Oct 30 13:06:49.765613 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:49.765585    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 13:06:49.765741 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:49.765617    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 13:06:49.792736 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:49.792710    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-6rj6c\""
Oct 30 13:06:49.819352 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:49.819324    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncd2g\" (UniqueName: \"kubernetes.io/projected/3d487ab7-4654-4757-ae7c-4250db7f2ef5-kube-api-access-ncd2g\") pod \"cert-manager-webhook-5dbbbf75cc-8269v\" (UID: \"3d487ab7-4654-4757-ae7c-4250db7f2ef5\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8269v"
Oct 30 13:06:49.819503 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:49.819421    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3d487ab7-4654-4757-ae7c-4250db7f2ef5-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-8269v\" (UID: \"3d487ab7-4654-4757-ae7c-4250db7f2ef5\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8269v"
Oct 30 13:06:49.920174 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:49.920119    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3d487ab7-4654-4757-ae7c-4250db7f2ef5-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-8269v\" (UID: \"3d487ab7-4654-4757-ae7c-4250db7f2ef5\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8269v"
Oct 30 13:06:49.920336 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:49.920199    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncd2g\" (UniqueName: \"kubernetes.io/projected/3d487ab7-4654-4757-ae7c-4250db7f2ef5-kube-api-access-ncd2g\") pod \"cert-manager-webhook-5dbbbf75cc-8269v\" (UID: \"3d487ab7-4654-4757-ae7c-4250db7f2ef5\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8269v"
Oct 30 13:06:49.941061 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:49.941031    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncd2g\" (UniqueName: \"kubernetes.io/projected/3d487ab7-4654-4757-ae7c-4250db7f2ef5-kube-api-access-ncd2g\") pod \"cert-manager-webhook-5dbbbf75cc-8269v\" (UID: \"3d487ab7-4654-4757-ae7c-4250db7f2ef5\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8269v"
Oct 30 13:06:49.941217 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:49.941107    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3d487ab7-4654-4757-ae7c-4250db7f2ef5-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-8269v\" (UID: \"3d487ab7-4654-4757-ae7c-4250db7f2ef5\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8269v"
Oct 30 13:06:50.062841 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:50.062758    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8269v"
Oct 30 13:06:50.378448 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:50.378419    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-8269v"]
Oct 30 13:06:50.380659 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:06:50.380633    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d487ab7_4654_4757_ae7c_4250db7f2ef5.slice/crio-73a61ffca657440a160165be8bb5ce3b6dc4dd238a433c803644c46aeddbf57c WatchSource:0}: Error finding container 73a61ffca657440a160165be8bb5ce3b6dc4dd238a433c803644c46aeddbf57c: Status 404 returned error can't find the container with id 73a61ffca657440a160165be8bb5ce3b6dc4dd238a433c803644c46aeddbf57c
Oct 30 13:06:51.172273 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:06:51.172241    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:06:51.270613 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:51.270575    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8269v" event={"ID":"3d487ab7-4654-4757-ae7c-4250db7f2ef5","Type":"ContainerStarted","Data":"73a61ffca657440a160165be8bb5ce3b6dc4dd238a433c803644c46aeddbf57c"}
Oct 30 13:06:53.281722 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:53.281682    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8269v" event={"ID":"3d487ab7-4654-4757-ae7c-4250db7f2ef5","Type":"ContainerStarted","Data":"fc6224256bde1989d48071c3f788204878de6c6816c0eed30e20ab841b55060c"}
Oct 30 13:06:53.282170 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:53.281770    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8269v"
Oct 30 13:06:53.326935 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:53.326884    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8269v" podStartSLOduration=1.7898927009999999 podStartE2EDuration="4.326868716s" totalImagesPullingTime="2.536976015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:53.324918937 +0000 UTC m=+232.472186178" watchObservedRunningTime="2026-10-30 13:06:53.326868716 +0000 UTC m=+232.474135956"
Oct 30 13:06:54.230770 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:54.230743    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-rd5mx"
Oct 30 13:06:58.723360 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:58.723326    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-ntjsh"]
Oct 30 13:06:58.726943 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:58.726925    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ntjsh"
Oct 30 13:06:58.761921 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:58.761886    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-nbggn\""
Oct 30 13:06:58.763715 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:58.763687    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-ntjsh"]
Oct 30 13:06:58.788958 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:58.788931    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 13:06:58.789094 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:58.788972    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 13:06:58.790285 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:58.790264    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnrdr\" (UniqueName: \"kubernetes.io/projected/8b8d6a8f-a170-4d6d-82f2-1d72828b55c5-kube-api-access-xnrdr\") pod \"authorino-operator-7b7bb4bc77-ntjsh\" (UID: \"8b8d6a8f-a170-4d6d-82f2-1d72828b55c5\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-ntjsh"
Oct 30 13:06:58.891222 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:58.891188    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnrdr\" (UniqueName: \"kubernetes.io/projected/8b8d6a8f-a170-4d6d-82f2-1d72828b55c5-kube-api-access-xnrdr\") pod \"authorino-operator-7b7bb4bc77-ntjsh\" (UID: \"8b8d6a8f-a170-4d6d-82f2-1d72828b55c5\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-ntjsh"
Oct 30 13:06:58.915273 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:58.915246    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnrdr\" (UniqueName: \"kubernetes.io/projected/8b8d6a8f-a170-4d6d-82f2-1d72828b55c5-kube-api-access-xnrdr\") pod \"authorino-operator-7b7bb4bc77-ntjsh\" (UID: \"8b8d6a8f-a170-4d6d-82f2-1d72828b55c5\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-ntjsh"
Oct 30 13:06:59.039401 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:59.039326    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ntjsh"
Oct 30 13:06:59.289299 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:59.289267    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8269v"
Oct 30 13:06:59.336443 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:06:59.336419    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-ntjsh"]
Oct 30 13:06:59.338937 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:06:59.338910    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b8d6a8f_a170_4d6d_82f2_1d72828b55c5.slice/crio-52412d559dd0c0e7d2b4aa4c4a2d23acaf46ae152d7dd99bce231f752cc3348d WatchSource:0}: Error finding container 52412d559dd0c0e7d2b4aa4c4a2d23acaf46ae152d7dd99bce231f752cc3348d: Status 404 returned error can't find the container with id 52412d559dd0c0e7d2b4aa4c4a2d23acaf46ae152d7dd99bce231f752cc3348d
Oct 30 13:07:00.321902 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:00.321858    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ntjsh" event={"ID":"8b8d6a8f-a170-4d6d-82f2-1d72828b55c5","Type":"ContainerStarted","Data":"52412d559dd0c0e7d2b4aa4c4a2d23acaf46ae152d7dd99bce231f752cc3348d"}
Oct 30 13:07:01.983066 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:01.983024    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-2fcf7"]
Oct 30 13:07:01.988086 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:01.988060    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-2fcf7"
Oct 30 13:07:02.019615 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:02.019588    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-89ksj\""
Oct 30 13:07:02.021305 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:02.021279    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-2fcf7"]
Oct 30 13:07:02.116342 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:02.116300    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xml6\" (UniqueName: \"kubernetes.io/projected/ad961809-9f3f-43dd-b97a-c4d6ebaa4457-kube-api-access-9xml6\") pod \"limitador-operator-controller-manager-5778bb4449-2fcf7\" (UID: \"ad961809-9f3f-43dd-b97a-c4d6ebaa4457\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-2fcf7"
Oct 30 13:07:02.217735 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:02.217691    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xml6\" (UniqueName: \"kubernetes.io/projected/ad961809-9f3f-43dd-b97a-c4d6ebaa4457-kube-api-access-9xml6\") pod \"limitador-operator-controller-manager-5778bb4449-2fcf7\" (UID: \"ad961809-9f3f-43dd-b97a-c4d6ebaa4457\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-2fcf7"
Oct 30 13:07:02.238814 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:02.238738    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xml6\" (UniqueName: \"kubernetes.io/projected/ad961809-9f3f-43dd-b97a-c4d6ebaa4457-kube-api-access-9xml6\") pod \"limitador-operator-controller-manager-5778bb4449-2fcf7\" (UID: \"ad961809-9f3f-43dd-b97a-c4d6ebaa4457\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-2fcf7"
Oct 30 13:07:02.300992 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:02.300957    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-2fcf7"
Oct 30 13:07:02.606955 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:02.606927    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-2fcf7"]
Oct 30 13:07:02.609014 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:07:02.608983    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad961809_9f3f_43dd_b97a_c4d6ebaa4457.slice/crio-89c3d52f2675b27ad0137b5b93c091b88f0e61698048e2353994790adadd36b2 WatchSource:0}: Error finding container 89c3d52f2675b27ad0137b5b93c091b88f0e61698048e2353994790adadd36b2: Status 404 returned error can't find the container with id 89c3d52f2675b27ad0137b5b93c091b88f0e61698048e2353994790adadd36b2
Oct 30 13:07:03.337887 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:03.337850    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-2fcf7" event={"ID":"ad961809-9f3f-43dd-b97a-c4d6ebaa4457","Type":"ContainerStarted","Data":"89c3d52f2675b27ad0137b5b93c091b88f0e61698048e2353994790adadd36b2"}
Oct 30 13:07:05.275671 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:05.275636    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-p4trh"]
Oct 30 13:07:05.279180 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:05.279156    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-p4trh"
Oct 30 13:07:05.312844 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:05.312819    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Oct 30 13:07:05.313019 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:05.312868    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-pwdvn\""
Oct 30 13:07:05.314652 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:05.314628    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-p4trh"]
Oct 30 13:07:05.344577 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:05.344491    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5fhs\" (UniqueName: \"kubernetes.io/projected/a955c0a1-e0ce-4ee8-8320-345bd556eff6-kube-api-access-p5fhs\") pod \"dns-operator-controller-manager-8557cbdb84-p4trh\" (UID: \"a955c0a1-e0ce-4ee8-8320-345bd556eff6\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-p4trh"
Oct 30 13:07:05.350262 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:05.350230    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-2fcf7" event={"ID":"ad961809-9f3f-43dd-b97a-c4d6ebaa4457","Type":"ContainerStarted","Data":"54ac00f2c21fc7993a17104837df074176bab665043418e163506f896fb4ef8c"}
Oct 30 13:07:05.350417 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:05.350295    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-2fcf7"
Oct 30 13:07:05.405191 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:05.405120    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-2fcf7" podStartSLOduration=1.9307842100000001 podStartE2EDuration="4.405105584s" totalImagesPullingTime="2.474321374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:05.396313389 +0000 UTC m=+244.543580629" watchObservedRunningTime="2026-10-30 13:07:05.405105584 +0000 UTC m=+244.552372825"
Oct 30 13:07:05.445940 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:05.445896    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5fhs\" (UniqueName: \"kubernetes.io/projected/a955c0a1-e0ce-4ee8-8320-345bd556eff6-kube-api-access-p5fhs\") pod \"dns-operator-controller-manager-8557cbdb84-p4trh\" (UID: \"a955c0a1-e0ce-4ee8-8320-345bd556eff6\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-p4trh"
Oct 30 13:07:05.469857 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:05.469825    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5fhs\" (UniqueName: \"kubernetes.io/projected/a955c0a1-e0ce-4ee8-8320-345bd556eff6-kube-api-access-p5fhs\") pod \"dns-operator-controller-manager-8557cbdb84-p4trh\" (UID: \"a955c0a1-e0ce-4ee8-8320-345bd556eff6\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-p4trh"
Oct 30 13:07:05.593014 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:05.592976    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-p4trh"
Oct 30 13:07:05.899958 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:05.899930    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-p4trh"]
Oct 30 13:07:05.901873 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:07:05.901840    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda955c0a1_e0ce_4ee8_8320_345bd556eff6.slice/crio-e627f32d60ed6662e6200c669ba5dc1ac8e9e52e76d89be4e018299f66137337 WatchSource:0}: Error finding container e627f32d60ed6662e6200c669ba5dc1ac8e9e52e76d89be4e018299f66137337: Status 404 returned error can't find the container with id e627f32d60ed6662e6200c669ba5dc1ac8e9e52e76d89be4e018299f66137337
Oct 30 13:07:06.356599 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:06.356562    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-p4trh" event={"ID":"a955c0a1-e0ce-4ee8-8320-345bd556eff6","Type":"ContainerStarted","Data":"e627f32d60ed6662e6200c669ba5dc1ac8e9e52e76d89be4e018299f66137337"}
Oct 30 13:07:08.370097 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:08.370058    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ntjsh" event={"ID":"8b8d6a8f-a170-4d6d-82f2-1d72828b55c5","Type":"ContainerStarted","Data":"41d6e8d40b3dcf99ed1712a8ee04fa43666fb0204f9493985a9efd82bb6178a0"}
Oct 30 13:07:08.370558 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:08.370275    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ntjsh"
Oct 30 13:07:08.415228 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:08.415174    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ntjsh" podStartSLOduration=2.135830865 podStartE2EDuration="10.41515781s" totalImagesPullingTime="8.279326945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:08.414361822 +0000 UTC m=+247.561629075" watchObservedRunningTime="2026-10-30 13:07:08.41515781 +0000 UTC m=+247.562425045"
Oct 30 13:07:09.376546 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:09.376505    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-p4trh" event={"ID":"a955c0a1-e0ce-4ee8-8320-345bd556eff6","Type":"ContainerStarted","Data":"c01f3c01b42dad8e39f8fecdb42856bfc8edb9b291232af1f5fccdd6e2dc8d6a"}
Oct 30 13:07:09.376993 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:09.376787    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-p4trh"
Oct 30 13:07:09.426064 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:09.425999    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-p4trh" podStartSLOduration=1.585485968 podStartE2EDuration="4.425983032s" totalImagesPullingTime="2.840497064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:09.422150114 +0000 UTC m=+248.569417349" watchObservedRunningTime="2026-10-30 13:07:09.425983032 +0000 UTC m=+248.573250273"
Oct 30 13:07:09.903101 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:09.903065    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zfv7q"]
Oct 30 13:07:09.906642 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:09.906623    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zfv7q"
Oct 30 13:07:09.919509 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:09.919470    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zfv7q"]
Oct 30 13:07:09.931399 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:09.931369    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-s9fs7\""
Oct 30 13:07:09.987290 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:09.987257    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x6r4\" (UniqueName: \"kubernetes.io/projected/fc701686-ddfa-4eca-be79-b8790e56d681-kube-api-access-5x6r4\") pod \"kuadrant-operator-controller-manager-678d87766f-zfv7q\" (UID: \"fc701686-ddfa-4eca-be79-b8790e56d681\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zfv7q"
Oct 30 13:07:09.987470 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:09.987301    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc701686-ddfa-4eca-be79-b8790e56d681-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-zfv7q\" (UID: \"fc701686-ddfa-4eca-be79-b8790e56d681\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zfv7q"
Oct 30 13:07:10.088700 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:10.088663    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x6r4\" (UniqueName: \"kubernetes.io/projected/fc701686-ddfa-4eca-be79-b8790e56d681-kube-api-access-5x6r4\") pod \"kuadrant-operator-controller-manager-678d87766f-zfv7q\" (UID: \"fc701686-ddfa-4eca-be79-b8790e56d681\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zfv7q"
Oct 30 13:07:10.088870 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:10.088712    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc701686-ddfa-4eca-be79-b8790e56d681-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-zfv7q\" (UID: \"fc701686-ddfa-4eca-be79-b8790e56d681\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zfv7q"
Oct 30 13:07:10.089088 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:10.089071    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc701686-ddfa-4eca-be79-b8790e56d681-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-zfv7q\" (UID: \"fc701686-ddfa-4eca-be79-b8790e56d681\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zfv7q"
Oct 30 13:07:10.110343 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:10.110310    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x6r4\" (UniqueName: \"kubernetes.io/projected/fc701686-ddfa-4eca-be79-b8790e56d681-kube-api-access-5x6r4\") pod \"kuadrant-operator-controller-manager-678d87766f-zfv7q\" (UID: \"fc701686-ddfa-4eca-be79-b8790e56d681\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zfv7q"
Oct 30 13:07:10.219811 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:10.219767    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zfv7q"
Oct 30 13:07:10.564403 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:10.564380    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zfv7q"]
Oct 30 13:07:10.565842 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:07:10.565811    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc701686_ddfa_4eca_be79_b8790e56d681.slice/crio-abbfa4ae92314f7b8b234d2936c6030d7b89baa4cca4ac52f030cd4d6c3ed99f WatchSource:0}: Error finding container abbfa4ae92314f7b8b234d2936c6030d7b89baa4cca4ac52f030cd4d6c3ed99f: Status 404 returned error can't find the container with id abbfa4ae92314f7b8b234d2936c6030d7b89baa4cca4ac52f030cd4d6c3ed99f
Oct 30 13:07:11.388881 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:11.388830    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zfv7q" event={"ID":"fc701686-ddfa-4eca-be79-b8790e56d681","Type":"ContainerStarted","Data":"abbfa4ae92314f7b8b234d2936c6030d7b89baa4cca4ac52f030cd4d6c3ed99f"}
Oct 30 13:07:15.417866 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:15.417828    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zfv7q" event={"ID":"fc701686-ddfa-4eca-be79-b8790e56d681","Type":"ContainerStarted","Data":"9cd2a5734394dde5ea620aeb20125ab5adcf87bdd133e4ccc8c8188d146b2fc7"}
Oct 30 13:07:15.418381 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:15.417974    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zfv7q"
Oct 30 13:07:15.489073 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:15.489017    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zfv7q" podStartSLOduration=2.75495187 podStartE2EDuration="6.488999797s" totalImagesPullingTime="3.734047927s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:15.473945686 +0000 UTC m=+254.621212926" watchObservedRunningTime="2026-10-30 13:07:15.488999797 +0000 UTC m=+254.636267037"
Oct 30 13:07:16.359277 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:16.359241    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-2fcf7"
Oct 30 13:07:17.377761 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:17.377719    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l"]
Oct 30 13:07:17.380955 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:17.380931    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l"
Oct 30 13:07:17.401512 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:17.401489    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-lc6n4\""
Oct 30 13:07:17.417293 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:17.417259    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l"]
Oct 30 13:07:17.453817 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:17.453784    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/142560f0-3d00-4d0c-9f87-58746dbe8a4e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l\" (UID: \"142560f0-3d00-4d0c-9f87-58746dbe8a4e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l"
Oct 30 13:07:17.453817 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:17.453818    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/142560f0-3d00-4d0c-9f87-58746dbe8a4e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l\" (UID: \"142560f0-3d00-4d0c-9f87-58746dbe8a4e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l"
Oct 30 13:07:17.454068 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:17.453838    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/142560f0-3d00-4d0c-9f87-58746dbe8a4e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l\" (UID: \"142560f0-3d00-4d0c-9f87-58746dbe8a4e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l"
Oct 30 13:07:17.454068 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:17.453870    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/142560f0-3d00-4d0c-9f87-58746dbe8a4e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l\" (UID: \"142560f0-3d00-4d0c-9f87-58746dbe8a4e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l"
Oct 30 13:07:17.454068 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:17.453961    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/142560f0-3d00-4d0c-9f87-58746dbe8a4e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l\" (UID: \"142560f0-3d00-4d0c-9f87-58746dbe8a4e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l"
Oct 30 13:07:17.454213 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:17.454082    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/142560f0-3d00-4d0c-9f87-58746dbe8a4e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l\" (UID: \"142560f0-3d00-4d0c-9f87-58746dbe8a4e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l"
Oct 30 13:07:17.454213 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:17.454116    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/142560f0-3d00-4d0c-9f87-58746dbe8a4e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l\" (UID: \"142560f0-3d00-4d0c-9f87-58746dbe8a4e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l"
Oct 30 13:07:17.454213 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:17.454200    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/142560f0-3d00-4d0c-9f87-58746dbe8a4e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l\" (UID: \"142560f0-3d00-4d0c-9f87-58746dbe8a4e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l"
Oct 30 13:07:17.454361 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:17.454293    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk9x8\" (UniqueName: \"kubernetes.io/projected/142560f0-3d00-4d0c-9f87-58746dbe8a4e-kube-api-access-hk9x8\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l\" (UID: \"142560f0-3d00-4d0c-9f87-58746dbe8a4e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l"
Oct 30 13:07:17.555219 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:17.555161    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/142560f0-3d00-4d0c-9f87-58746dbe8a4e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l\" (UID: \"142560f0-3d00-4d0c-9f87-58746dbe8a4e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l"
Oct 30 13:07:17.555219 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:17.555220    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/142560f0-3d00-4d0c-9f87-58746dbe8a4e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l\" (UID: \"142560f0-3d00-4d0c-9f87-58746dbe8a4e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l"
Oct 30 13:07:17.555490 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:17.555239    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/142560f0-3d00-4d0c-9f87-58746dbe8a4e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l\" (UID: \"142560f0-3d00-4d0c-9f87-58746dbe8a4e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l"
Oct 30 13:07:17.555490 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:17.555274    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/142560f0-3d00-4d0c-9f87-58746dbe8a4e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l\" (UID: \"142560f0-3d00-4d0c-9f87-58746dbe8a4e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l"
Oct 30 13:07:17.555490 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:17.555293    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/142560f0-3d00-4d0c-9f87-58746dbe8a4e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l\" (UID: \"142560f0-3d00-4d0c-9f87-58746dbe8a4e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l"
Oct 30 13:07:17.555490 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:17.555322    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/142560f0-3d00-4d0c-9f87-58746dbe8a4e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l\" (UID: \"142560f0-3d00-4d0c-9f87-58746dbe8a4e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l"
Oct 30 13:07:17.555490 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:17.555349    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/142560f0-3d00-4d0c-9f87-58746dbe8a4e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l\" (UID: \"142560f0-3d00-4d0c-9f87-58746dbe8a4e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l"
Oct 30 13:07:17.555490 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:17.555418    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/142560f0-3d00-4d0c-9f87-58746dbe8a4e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l\" (UID: \"142560f0-3d00-4d0c-9f87-58746dbe8a4e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l"
Oct 30 13:07:17.555490 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:17.555451    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk9x8\" (UniqueName: \"kubernetes.io/projected/142560f0-3d00-4d0c-9f87-58746dbe8a4e-kube-api-access-hk9x8\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l\" (UID: \"142560f0-3d00-4d0c-9f87-58746dbe8a4e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l"
Oct 30 13:07:17.555852 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:17.555794    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/142560f0-3d00-4d0c-9f87-58746dbe8a4e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l\" (UID: \"142560f0-3d00-4d0c-9f87-58746dbe8a4e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l"
Oct 30 13:07:17.555852 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:17.555791    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/142560f0-3d00-4d0c-9f87-58746dbe8a4e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l\" (UID: \"142560f0-3d00-4d0c-9f87-58746dbe8a4e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l"
Oct 30 13:07:17.556004 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:17.555973    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/142560f0-3d00-4d0c-9f87-58746dbe8a4e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l\" (UID: \"142560f0-3d00-4d0c-9f87-58746dbe8a4e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l"
Oct 30 13:07:17.556154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:17.556053    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/142560f0-3d00-4d0c-9f87-58746dbe8a4e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l\" (UID: \"142560f0-3d00-4d0c-9f87-58746dbe8a4e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l"
Oct 30 13:07:17.556393 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:17.556371    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/142560f0-3d00-4d0c-9f87-58746dbe8a4e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l\" (UID: \"142560f0-3d00-4d0c-9f87-58746dbe8a4e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l"
Oct 30 13:07:17.557872 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:17.557838    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/142560f0-3d00-4d0c-9f87-58746dbe8a4e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l\" (UID: \"142560f0-3d00-4d0c-9f87-58746dbe8a4e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l"
Oct 30 13:07:17.558078 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:17.558059    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/142560f0-3d00-4d0c-9f87-58746dbe8a4e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l\" (UID: \"142560f0-3d00-4d0c-9f87-58746dbe8a4e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l"
Oct 30 13:07:17.578710 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:17.578680    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/142560f0-3d00-4d0c-9f87-58746dbe8a4e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l\" (UID: \"142560f0-3d00-4d0c-9f87-58746dbe8a4e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l"
Oct 30 13:07:17.578843 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:17.578797    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk9x8\" (UniqueName: \"kubernetes.io/projected/142560f0-3d00-4d0c-9f87-58746dbe8a4e-kube-api-access-hk9x8\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l\" (UID: \"142560f0-3d00-4d0c-9f87-58746dbe8a4e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l"
Oct 30 13:07:17.695512 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:17.695466    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l"
Oct 30 13:07:18.025971 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:18.025943    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l"]
Oct 30 13:07:18.028546 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:07:18.028517    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod142560f0_3d00_4d0c_9f87_58746dbe8a4e.slice/crio-d9f63abec03ca132168af3eb8683f517edc4dc45a378a950d863ccb3708dc5dd WatchSource:0}: Error finding container d9f63abec03ca132168af3eb8683f517edc4dc45a378a950d863ccb3708dc5dd: Status 404 returned error can't find the container with id d9f63abec03ca132168af3eb8683f517edc4dc45a378a950d863ccb3708dc5dd
Oct 30 13:07:18.030949 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:18.030887    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:07:18.031061 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:18.030988    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:07:18.031061 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:18.031045    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:07:18.436049 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:18.436011    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l" event={"ID":"142560f0-3d00-4d0c-9f87-58746dbe8a4e","Type":"ContainerStarted","Data":"ccb72ecacea8d6da91cf218fb2333bde7b092622e8196bf1f3b8e5cc2d661678"}
Oct 30 13:07:18.436462 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:18.436070    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l" event={"ID":"142560f0-3d00-4d0c-9f87-58746dbe8a4e","Type":"ContainerStarted","Data":"d9f63abec03ca132168af3eb8683f517edc4dc45a378a950d863ccb3708dc5dd"}
Oct 30 13:07:18.496341 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:18.496286    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l" podStartSLOduration=1.496270838 podStartE2EDuration="1.496270838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:18.493382004 +0000 UTC m=+257.640649244" watchObservedRunningTime="2026-10-30 13:07:18.496270838 +0000 UTC m=+257.643538078"
Oct 30 13:07:18.695604 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:18.695525    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l"
Oct 30 13:07:19.378415 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:19.378380    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ntjsh"
Oct 30 13:07:19.690049 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:19.690014    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-97tc8"]
Oct 30 13:07:19.693538 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:19.693522    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-97tc8"
Oct 30 13:07:19.702218 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:19.702194    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l"
Oct 30 13:07:19.706924 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:19.706897    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-97tc8"]
Oct 30 13:07:19.746353 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:19.746322    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-sxnrz\""
Oct 30 13:07:19.774469 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:19.774431    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v67l\" (UniqueName: \"kubernetes.io/projected/9292a1bf-e6c2-4dd0-bdf6-a04fb0f97226-kube-api-access-4v67l\") pod \"authorino-57f7f8659f-97tc8\" (UID: \"9292a1bf-e6c2-4dd0-bdf6-a04fb0f97226\") " pod="kuadrant-system/authorino-57f7f8659f-97tc8"
Oct 30 13:07:19.875615 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:19.875577    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v67l\" (UniqueName: \"kubernetes.io/projected/9292a1bf-e6c2-4dd0-bdf6-a04fb0f97226-kube-api-access-4v67l\") pod \"authorino-57f7f8659f-97tc8\" (UID: \"9292a1bf-e6c2-4dd0-bdf6-a04fb0f97226\") " pod="kuadrant-system/authorino-57f7f8659f-97tc8"
Oct 30 13:07:19.904766 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:19.904723    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v67l\" (UniqueName: \"kubernetes.io/projected/9292a1bf-e6c2-4dd0-bdf6-a04fb0f97226-kube-api-access-4v67l\") pod \"authorino-57f7f8659f-97tc8\" (UID: \"9292a1bf-e6c2-4dd0-bdf6-a04fb0f97226\") " pod="kuadrant-system/authorino-57f7f8659f-97tc8"
Oct 30 13:07:20.007584 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:20.007491    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-97tc8"
Oct 30 13:07:20.304601 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:20.304572    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-97tc8"]
Oct 30 13:07:20.306517 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:07:20.306486    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9292a1bf_e6c2_4dd0_bdf6_a04fb0f97226.slice/crio-9ed30635ad8254cffdc59932c6f1b4ad3aeb61999d424d948f7ac18f6f0c6ca8 WatchSource:0}: Error finding container 9ed30635ad8254cffdc59932c6f1b4ad3aeb61999d424d948f7ac18f6f0c6ca8: Status 404 returned error can't find the container with id 9ed30635ad8254cffdc59932c6f1b4ad3aeb61999d424d948f7ac18f6f0c6ca8
Oct 30 13:07:20.383818 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:20.383786    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-p4trh"
Oct 30 13:07:20.450317 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:20.450281    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-97tc8" event={"ID":"9292a1bf-e6c2-4dd0-bdf6-a04fb0f97226","Type":"ContainerStarted","Data":"9ed30635ad8254cffdc59932c6f1b4ad3aeb61999d424d948f7ac18f6f0c6ca8"}
Oct 30 13:07:20.450610 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:20.450578    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l"
Oct 30 13:07:20.451722 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:20.451699    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l"
Oct 30 13:07:21.167853 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:07:21.167815    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:07:23.469120 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:23.469072    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-97tc8" event={"ID":"9292a1bf-e6c2-4dd0-bdf6-a04fb0f97226","Type":"ContainerStarted","Data":"295d49e9764868372634a30bebc10911319f9756be3871320c0493798446b42a"}
Oct 30 13:07:23.533352 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:23.533241    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-97tc8" podStartSLOduration=1.6064369219999999 podStartE2EDuration="4.533226072s" totalImagesPullingTime="2.92678915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:23.523072835 +0000 UTC m=+262.670340076" watchObservedRunningTime="2026-10-30 13:07:23.533226072 +0000 UTC m=+262.680493312"
Oct 30 13:07:24.597612 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:24.597579    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-865545ddfd-nqrbp"]
Oct 30 13:07:24.601209 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:24.601186    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-865545ddfd-nqrbp"
Oct 30 13:07:24.618728 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:24.618703    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-865545ddfd-nqrbp"]
Oct 30 13:07:24.632233 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:24.632199    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 30 13:07:24.632403 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:24.632250    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Oct 30 13:07:24.632403 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:24.632259    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Oct 30 13:07:24.632403 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:24.632284    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Oct 30 13:07:24.632403 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:24.632311    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-qqqdr\""
Oct 30 13:07:24.716356 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:24.716304    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/40296c3c-8fc1-488f-9bae-51273b5fba28-manager-config\") pod \"jobset-controller-manager-865545ddfd-nqrbp\" (UID: \"40296c3c-8fc1-488f-9bae-51273b5fba28\") " pod="openshift-jobset-operator/jobset-controller-manager-865545ddfd-nqrbp"
Oct 30 13:07:24.716535 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:24.716368    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/40296c3c-8fc1-488f-9bae-51273b5fba28-cert\") pod \"jobset-controller-manager-865545ddfd-nqrbp\" (UID: \"40296c3c-8fc1-488f-9bae-51273b5fba28\") " pod="openshift-jobset-operator/jobset-controller-manager-865545ddfd-nqrbp"
Oct 30 13:07:24.716535 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:24.716460    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40296c3c-8fc1-488f-9bae-51273b5fba28-metrics-certs\") pod \"jobset-controller-manager-865545ddfd-nqrbp\" (UID: \"40296c3c-8fc1-488f-9bae-51273b5fba28\") " pod="openshift-jobset-operator/jobset-controller-manager-865545ddfd-nqrbp"
Oct 30 13:07:24.716535 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:24.716495    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pjkh\" (UniqueName: \"kubernetes.io/projected/40296c3c-8fc1-488f-9bae-51273b5fba28-kube-api-access-4pjkh\") pod \"jobset-controller-manager-865545ddfd-nqrbp\" (UID: \"40296c3c-8fc1-488f-9bae-51273b5fba28\") " pod="openshift-jobset-operator/jobset-controller-manager-865545ddfd-nqrbp"
Oct 30 13:07:24.718007 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:24.717985    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 30 13:07:24.817417 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:24.817368    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/40296c3c-8fc1-488f-9bae-51273b5fba28-manager-config\") pod \"jobset-controller-manager-865545ddfd-nqrbp\" (UID: \"40296c3c-8fc1-488f-9bae-51273b5fba28\") " pod="openshift-jobset-operator/jobset-controller-manager-865545ddfd-nqrbp"
Oct 30 13:07:24.817417 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:24.817422    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/40296c3c-8fc1-488f-9bae-51273b5fba28-cert\") pod \"jobset-controller-manager-865545ddfd-nqrbp\" (UID: \"40296c3c-8fc1-488f-9bae-51273b5fba28\") " pod="openshift-jobset-operator/jobset-controller-manager-865545ddfd-nqrbp"
Oct 30 13:07:24.817690 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:24.817476    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40296c3c-8fc1-488f-9bae-51273b5fba28-metrics-certs\") pod \"jobset-controller-manager-865545ddfd-nqrbp\" (UID: \"40296c3c-8fc1-488f-9bae-51273b5fba28\") " pod="openshift-jobset-operator/jobset-controller-manager-865545ddfd-nqrbp"
Oct 30 13:07:24.817690 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:24.817522    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pjkh\" (UniqueName: \"kubernetes.io/projected/40296c3c-8fc1-488f-9bae-51273b5fba28-kube-api-access-4pjkh\") pod \"jobset-controller-manager-865545ddfd-nqrbp\" (UID: \"40296c3c-8fc1-488f-9bae-51273b5fba28\") " pod="openshift-jobset-operator/jobset-controller-manager-865545ddfd-nqrbp"
Oct 30 13:07:24.818330 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:24.818307    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/40296c3c-8fc1-488f-9bae-51273b5fba28-manager-config\") pod \"jobset-controller-manager-865545ddfd-nqrbp\" (UID: \"40296c3c-8fc1-488f-9bae-51273b5fba28\") " pod="openshift-jobset-operator/jobset-controller-manager-865545ddfd-nqrbp"
Oct 30 13:07:24.820296 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:24.820261    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/40296c3c-8fc1-488f-9bae-51273b5fba28-cert\") pod \"jobset-controller-manager-865545ddfd-nqrbp\" (UID: \"40296c3c-8fc1-488f-9bae-51273b5fba28\") " pod="openshift-jobset-operator/jobset-controller-manager-865545ddfd-nqrbp"
Oct 30 13:07:24.820391 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:24.820364    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40296c3c-8fc1-488f-9bae-51273b5fba28-metrics-certs\") pod \"jobset-controller-manager-865545ddfd-nqrbp\" (UID: \"40296c3c-8fc1-488f-9bae-51273b5fba28\") " pod="openshift-jobset-operator/jobset-controller-manager-865545ddfd-nqrbp"
Oct 30 13:07:24.842227 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:24.842197    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pjkh\" (UniqueName: \"kubernetes.io/projected/40296c3c-8fc1-488f-9bae-51273b5fba28-kube-api-access-4pjkh\") pod \"jobset-controller-manager-865545ddfd-nqrbp\" (UID: \"40296c3c-8fc1-488f-9bae-51273b5fba28\") " pod="openshift-jobset-operator/jobset-controller-manager-865545ddfd-nqrbp"
Oct 30 13:07:24.913823 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:24.913732    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-865545ddfd-nqrbp"
Oct 30 13:07:25.050397 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:25.050365    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-97tc8"]
Oct 30 13:07:25.216078 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:25.216051    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-865545ddfd-nqrbp"]
Oct 30 13:07:25.218312 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:07:25.218269    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40296c3c_8fc1_488f_9bae_51273b5fba28.slice/crio-4cc76c5eacd3744d80d8be037eb6a4aab6b817e33772faca416888a9cbc0b99e WatchSource:0}: Error finding container 4cc76c5eacd3744d80d8be037eb6a4aab6b817e33772faca416888a9cbc0b99e: Status 404 returned error can't find the container with id 4cc76c5eacd3744d80d8be037eb6a4aab6b817e33772faca416888a9cbc0b99e
Oct 30 13:07:25.480085 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:25.479998    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-865545ddfd-nqrbp" event={"ID":"40296c3c-8fc1-488f-9bae-51273b5fba28","Type":"ContainerStarted","Data":"4cc76c5eacd3744d80d8be037eb6a4aab6b817e33772faca416888a9cbc0b99e"}
Oct 30 13:07:25.480262 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:25.480098    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-97tc8" podUID="9292a1bf-e6c2-4dd0-bdf6-a04fb0f97226" containerName="authorino" containerID="cri-o://295d49e9764868372634a30bebc10911319f9756be3871320c0493798446b42a" gracePeriod=30
Oct 30 13:07:25.713029 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:25.713000    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-97tc8"
Oct 30 13:07:25.825974 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:25.825894    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9292a1bf-e6c2-4dd0-bdf6-a04fb0f97226-kube-api-access-4v67l\" (UniqueName: \"kubernetes.io/projected/9292a1bf-e6c2-4dd0-bdf6-a04fb0f97226-kube-api-access-4v67l\") pod \"9292a1bf-e6c2-4dd0-bdf6-a04fb0f97226\" (UID: \"9292a1bf-e6c2-4dd0-bdf6-a04fb0f97226\") "
Oct 30 13:07:25.828575 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:25.828547    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9292a1bf-e6c2-4dd0-bdf6-a04fb0f97226-kube-api-access-4v67l" pod "9292a1bf-e6c2-4dd0-bdf6-a04fb0f97226" (UID: "9292a1bf-e6c2-4dd0-bdf6-a04fb0f97226"). InnerVolumeSpecName "kube-api-access-4v67l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:25.927021 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:25.926969    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v67l\" (UniqueName: \"kubernetes.io/projected/9292a1bf-e6c2-4dd0-bdf6-a04fb0f97226-kube-api-access-4v67l\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:07:26.427314 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:26.427279    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zfv7q"
Oct 30 13:07:26.497270 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:26.495615    5093 generic.go:356] "Generic (PLEG): container finished" podID="9292a1bf-e6c2-4dd0-bdf6-a04fb0f97226" containerID="295d49e9764868372634a30bebc10911319f9756be3871320c0493798446b42a" exitCode=0
Oct 30 13:07:26.497270 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:26.495762    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-97tc8" event={"ID":"9292a1bf-e6c2-4dd0-bdf6-a04fb0f97226","Type":"ContainerDied","Data":"295d49e9764868372634a30bebc10911319f9756be3871320c0493798446b42a"}
Oct 30 13:07:26.497270 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:26.495801    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-97tc8" event={"ID":"9292a1bf-e6c2-4dd0-bdf6-a04fb0f97226","Type":"ContainerDied","Data":"9ed30635ad8254cffdc59932c6f1b4ad3aeb61999d424d948f7ac18f6f0c6ca8"}
Oct 30 13:07:26.497270 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:26.495824    5093 scope.go:122] "RemoveContainer" containerID="295d49e9764868372634a30bebc10911319f9756be3871320c0493798446b42a"
Oct 30 13:07:26.497270 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:26.495978    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-97tc8"
Oct 30 13:07:26.522335 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:26.522306    5093 scope.go:122] "RemoveContainer" containerID="295d49e9764868372634a30bebc10911319f9756be3871320c0493798446b42a"
Oct 30 13:07:26.523336 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:07:26.523307    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"295d49e9764868372634a30bebc10911319f9756be3871320c0493798446b42a\": container with ID starting with 295d49e9764868372634a30bebc10911319f9756be3871320c0493798446b42a not found: ID does not exist" containerID="295d49e9764868372634a30bebc10911319f9756be3871320c0493798446b42a"
Oct 30 13:07:26.523441 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:26.523368    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"295d49e9764868372634a30bebc10911319f9756be3871320c0493798446b42a"} err="failed to get container status \"295d49e9764868372634a30bebc10911319f9756be3871320c0493798446b42a\": rpc error: code = NotFound desc = could not find container \"295d49e9764868372634a30bebc10911319f9756be3871320c0493798446b42a\": container with ID starting with 295d49e9764868372634a30bebc10911319f9756be3871320c0493798446b42a not found: ID does not exist"
Oct 30 13:07:26.564739 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:26.564701    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-97tc8"]
Oct 30 13:07:26.568895 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:26.568832    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-97tc8"]
Oct 30 13:07:26.998827 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:26.998788    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9292a1bf-e6c2-4dd0-bdf6-a04fb0f97226" path="/var/lib/kubelet/pods/9292a1bf-e6c2-4dd0-bdf6-a04fb0f97226/volumes"
Oct 30 13:07:28.510064 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:28.510025    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-865545ddfd-nqrbp" event={"ID":"40296c3c-8fc1-488f-9bae-51273b5fba28","Type":"ContainerStarted","Data":"06f5d82dd2925c8beaeab483bdf594b2be838204eac29509e622d9893d90cba9"}
Oct 30 13:07:28.510535 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:28.510191    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-865545ddfd-nqrbp"
Oct 30 13:07:28.556532 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:28.556474    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-865545ddfd-nqrbp" podStartSLOduration=1.794668838 podStartE2EDuration="4.556460766s" totalImagesPullingTime="2.761791928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:28.554947768 +0000 UTC m=+267.702215008" watchObservedRunningTime="2026-10-30 13:07:28.556460766 +0000 UTC m=+267.703728006"
Oct 30 13:07:39.520812 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:07:39.520773    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-865545ddfd-nqrbp"
Oct 30 13:07:51.165599 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:07:51.165558    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:08:01.034740 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:01.034706    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jdj4_f93ee017-ab79-447c-90e0-5b85b71dba24/console-operator/1.log"
Oct 30 13:08:01.035970 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:01.035944    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jdj4_f93ee017-ab79-447c-90e0-5b85b71dba24/console-operator/1.log"
Oct 30 13:08:01.040105 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:01.040085    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tnm_3a675ac7-fbe1-449b-accf-bb0456ed0bd3/ovn-acl-logging/0.log"
Oct 30 13:08:01.041214 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:01.041196    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tnm_3a675ac7-fbe1-449b-accf-bb0456ed0bd3/ovn-acl-logging/0.log"
Oct 30 13:08:21.165513 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:08:21.165470    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:08:23.175827 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:23.175788    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-k8hb7"]
Oct 30 13:08:23.176252 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:23.176215    5093 cpu_manager.go:414] "containerMap: removing container" podUID="9292a1bf-e6c2-4dd0-bdf6-a04fb0f97226" containerName="authorino"
Oct 30 13:08:23.176252 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:23.176228    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9292a1bf-e6c2-4dd0-bdf6-a04fb0f97226" containerName="authorino"
Oct 30 13:08:23.176336 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:23.176300    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="9292a1bf-e6c2-4dd0-bdf6-a04fb0f97226" containerName="authorino"
Oct 30 13:08:23.179479 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:23.179460    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-k8hb7"
Oct 30 13:08:23.227469 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:23.227427    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 13:08:23.227469 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:23.227462    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 13:08:23.227469 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:23.227479    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-w9b4k\""
Oct 30 13:08:23.227704 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:23.227504    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Oct 30 13:08:23.229566 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:23.229543    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-k8hb7"]
Oct 30 13:08:23.249301 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:23.249269    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-hdnsr"]
Oct 30 13:08:23.253223 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:23.253196    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-hdnsr"
Oct 30 13:08:23.297554 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:23.297491    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-hdnsr"]
Oct 30 13:08:23.309580 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:23.309545    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Oct 30 13:08:23.309755 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:23.309622    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Oct 30 13:08:23.309755 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:23.309630    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-ggvrm\""
Oct 30 13:08:23.327103 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:23.327072    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w55dr\" (UniqueName: \"kubernetes.io/projected/775251e0-43d1-4239-a077-1b9fc7e56fe4-kube-api-access-w55dr\") pod \"odh-model-controller-769875db56-hdnsr\" (UID: \"775251e0-43d1-4239-a077-1b9fc7e56fe4\") " pod="opendatahub/odh-model-controller-769875db56-hdnsr"
Oct 30 13:08:23.327276 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:23.327117    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/775251e0-43d1-4239-a077-1b9fc7e56fe4-metrics-certs\") pod \"odh-model-controller-769875db56-hdnsr\" (UID: \"775251e0-43d1-4239-a077-1b9fc7e56fe4\") " pod="opendatahub/odh-model-controller-769875db56-hdnsr"
Oct 30 13:08:23.327276 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:23.327248    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/775251e0-43d1-4239-a077-1b9fc7e56fe4-cert\") pod \"odh-model-controller-769875db56-hdnsr\" (UID: \"775251e0-43d1-4239-a077-1b9fc7e56fe4\") " pod="opendatahub/odh-model-controller-769875db56-hdnsr"
Oct 30 13:08:23.327376 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:23.327279    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcn2z\" (UniqueName: \"kubernetes.io/projected/ebddb579-1088-4d8f-8917-e7e14883534a-kube-api-access-qcn2z\") pod \"model-serving-api-75bd89fdfc-k8hb7\" (UID: \"ebddb579-1088-4d8f-8917-e7e14883534a\") " pod="opendatahub/model-serving-api-75bd89fdfc-k8hb7"
Oct 30 13:08:23.327376 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:23.327325    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebddb579-1088-4d8f-8917-e7e14883534a-tls-certs\") pod \"model-serving-api-75bd89fdfc-k8hb7\" (UID: \"ebddb579-1088-4d8f-8917-e7e14883534a\") " pod="opendatahub/model-serving-api-75bd89fdfc-k8hb7"
Oct 30 13:08:23.428539 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:23.428454    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebddb579-1088-4d8f-8917-e7e14883534a-tls-certs\") pod \"model-serving-api-75bd89fdfc-k8hb7\" (UID: \"ebddb579-1088-4d8f-8917-e7e14883534a\") " pod="opendatahub/model-serving-api-75bd89fdfc-k8hb7"
Oct 30 13:08:23.428539 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:23.428497    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w55dr\" (UniqueName: \"kubernetes.io/projected/775251e0-43d1-4239-a077-1b9fc7e56fe4-kube-api-access-w55dr\") pod \"odh-model-controller-769875db56-hdnsr\" (UID: \"775251e0-43d1-4239-a077-1b9fc7e56fe4\") " pod="opendatahub/odh-model-controller-769875db56-hdnsr"
Oct 30 13:08:23.428539 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:23.428529    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/775251e0-43d1-4239-a077-1b9fc7e56fe4-metrics-certs\") pod \"odh-model-controller-769875db56-hdnsr\" (UID: \"775251e0-43d1-4239-a077-1b9fc7e56fe4\") " pod="opendatahub/odh-model-controller-769875db56-hdnsr"
Oct 30 13:08:23.428829 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:23.428572    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/775251e0-43d1-4239-a077-1b9fc7e56fe4-cert\") pod \"odh-model-controller-769875db56-hdnsr\" (UID: \"775251e0-43d1-4239-a077-1b9fc7e56fe4\") " pod="opendatahub/odh-model-controller-769875db56-hdnsr"
Oct 30 13:08:23.428829 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:23.428591    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcn2z\" (UniqueName: \"kubernetes.io/projected/ebddb579-1088-4d8f-8917-e7e14883534a-kube-api-access-qcn2z\") pod \"model-serving-api-75bd89fdfc-k8hb7\" (UID: \"ebddb579-1088-4d8f-8917-e7e14883534a\") " pod="opendatahub/model-serving-api-75bd89fdfc-k8hb7"
Oct 30 13:08:23.431108 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:23.431074    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/775251e0-43d1-4239-a077-1b9fc7e56fe4-cert\") pod \"odh-model-controller-769875db56-hdnsr\" (UID: \"775251e0-43d1-4239-a077-1b9fc7e56fe4\") " pod="opendatahub/odh-model-controller-769875db56-hdnsr"
Oct 30 13:08:23.431242 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:23.431084    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/775251e0-43d1-4239-a077-1b9fc7e56fe4-metrics-certs\") pod \"odh-model-controller-769875db56-hdnsr\" (UID: \"775251e0-43d1-4239-a077-1b9fc7e56fe4\") " pod="opendatahub/odh-model-controller-769875db56-hdnsr"
Oct 30 13:08:23.431242 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:23.431210    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebddb579-1088-4d8f-8917-e7e14883534a-tls-certs\") pod \"model-serving-api-75bd89fdfc-k8hb7\" (UID: \"ebddb579-1088-4d8f-8917-e7e14883534a\") " pod="opendatahub/model-serving-api-75bd89fdfc-k8hb7"
Oct 30 13:08:23.450245 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:23.450203    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w55dr\" (UniqueName: \"kubernetes.io/projected/775251e0-43d1-4239-a077-1b9fc7e56fe4-kube-api-access-w55dr\") pod \"odh-model-controller-769875db56-hdnsr\" (UID: \"775251e0-43d1-4239-a077-1b9fc7e56fe4\") " pod="opendatahub/odh-model-controller-769875db56-hdnsr"
Oct 30 13:08:23.450641 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:23.450613    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcn2z\" (UniqueName: \"kubernetes.io/projected/ebddb579-1088-4d8f-8917-e7e14883534a-kube-api-access-qcn2z\") pod \"model-serving-api-75bd89fdfc-k8hb7\" (UID: \"ebddb579-1088-4d8f-8917-e7e14883534a\") " pod="opendatahub/model-serving-api-75bd89fdfc-k8hb7"
Oct 30 13:08:23.492611 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:23.492558    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-k8hb7"
Oct 30 13:08:23.565746 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:23.565709    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-hdnsr"
Oct 30 13:08:23.789240 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:23.789208    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-k8hb7"]
Oct 30 13:08:23.791053 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:08:23.791023    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebddb579_1088_4d8f_8917_e7e14883534a.slice/crio-a34f64a43029350828eaf4312a578fb5aea8df11b76fe8610fb394a42294bd45 WatchSource:0}: Error finding container a34f64a43029350828eaf4312a578fb5aea8df11b76fe8610fb394a42294bd45: Status 404 returned error can't find the container with id a34f64a43029350828eaf4312a578fb5aea8df11b76fe8610fb394a42294bd45
Oct 30 13:08:23.792853 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:23.792835    5093 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 13:08:23.862884 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:23.862725    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-hdnsr"]
Oct 30 13:08:23.865032 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:08:23.865004    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod775251e0_43d1_4239_a077_1b9fc7e56fe4.slice/crio-35338d6d5b2177d92bac792cea470786636aed7a1b368b36c561c2e5070eda2d WatchSource:0}: Error finding container 35338d6d5b2177d92bac792cea470786636aed7a1b368b36c561c2e5070eda2d: Status 404 returned error can't find the container with id 35338d6d5b2177d92bac792cea470786636aed7a1b368b36c561c2e5070eda2d
Oct 30 13:08:24.801214 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:24.801172    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-k8hb7" event={"ID":"ebddb579-1088-4d8f-8917-e7e14883534a","Type":"ContainerStarted","Data":"a34f64a43029350828eaf4312a578fb5aea8df11b76fe8610fb394a42294bd45"}
Oct 30 13:08:24.802767 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:24.802730    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-hdnsr" event={"ID":"775251e0-43d1-4239-a077-1b9fc7e56fe4","Type":"ContainerStarted","Data":"35338d6d5b2177d92bac792cea470786636aed7a1b368b36c561c2e5070eda2d"}
Oct 30 13:08:26.816044 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:26.815936    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-k8hb7" event={"ID":"ebddb579-1088-4d8f-8917-e7e14883534a","Type":"ContainerStarted","Data":"372732a5eb92024e221a5b46219a98d13d9b87d488701c9bae58b951903003a6"}
Oct 30 13:08:26.816551 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:26.816154    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-k8hb7"
Oct 30 13:08:26.818200 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:26.818171    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-hdnsr" event={"ID":"775251e0-43d1-4239-a077-1b9fc7e56fe4","Type":"ContainerStarted","Data":"68b5e204065bec969e1aad84bcfecfc3eb1d7d8fcd113ea6688aacde69312446"}
Oct 30 13:08:26.818361 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:26.818230    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-hdnsr"
Oct 30 13:08:26.859733 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:26.859681    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-k8hb7" podStartSLOduration=1.7955445399999999 podStartE2EDuration="3.859667706s" totalImagesPullingTime="2.064123166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:26.857718679 +0000 UTC m=+326.004985923" watchObservedRunningTime="2026-10-30 13:08:26.859667706 +0000 UTC m=+326.006934946"
Oct 30 13:08:26.902316 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:26.902267    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-hdnsr" podStartSLOduration=1.211107293 podStartE2EDuration="3.902251805s" totalImagesPullingTime="2.691144512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:26.901223856 +0000 UTC m=+326.048491092" watchObservedRunningTime="2026-10-30 13:08:26.902251805 +0000 UTC m=+326.049519045"
Oct 30 13:08:37.826589 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:37.826555    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-hdnsr"
Oct 30 13:08:37.827764 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:37.827741    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-k8hb7"
Oct 30 13:08:45.290885 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:45.290841    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-n6nk4"]
Oct 30 13:08:45.294590 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:45.294567    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-n6nk4"
Oct 30 13:08:45.309532 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:45.309500    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-n6nk4"]
Oct 30 13:08:45.320376 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:45.320344    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-sxnrz\""
Oct 30 13:08:45.328615 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:45.328586    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsx5h\" (UniqueName: \"kubernetes.io/projected/e04e348c-99c2-419c-8213-365c6607d27f-kube-api-access-zsx5h\") pod \"authorino-6f6944bcdd-n6nk4\" (UID: \"e04e348c-99c2-419c-8213-365c6607d27f\") " pod="kuadrant-system/authorino-6f6944bcdd-n6nk4"
Oct 30 13:08:45.429567 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:45.429508    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsx5h\" (UniqueName: \"kubernetes.io/projected/e04e348c-99c2-419c-8213-365c6607d27f-kube-api-access-zsx5h\") pod \"authorino-6f6944bcdd-n6nk4\" (UID: \"e04e348c-99c2-419c-8213-365c6607d27f\") " pod="kuadrant-system/authorino-6f6944bcdd-n6nk4"
Oct 30 13:08:45.450340 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:45.450299    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsx5h\" (UniqueName: \"kubernetes.io/projected/e04e348c-99c2-419c-8213-365c6607d27f-kube-api-access-zsx5h\") pod \"authorino-6f6944bcdd-n6nk4\" (UID: \"e04e348c-99c2-419c-8213-365c6607d27f\") " pod="kuadrant-system/authorino-6f6944bcdd-n6nk4"
Oct 30 13:08:45.606869 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:45.606787    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-n6nk4"
Oct 30 13:08:45.607045 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:45.607020    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-n6nk4"]
Oct 30 13:08:45.635540 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:45.635505    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-w6w7p"]
Oct 30 13:08:45.639467 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:45.639440    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-w6w7p"
Oct 30 13:08:45.655018 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:45.654992    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 30 13:08:45.655551 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:45.655522    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-w6w7p"]
Oct 30 13:08:45.684356 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:45.684247    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-w6w7p"]
Oct 30 13:08:45.684581 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:08:45.684516    5093 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-kwdk8 tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-kwdk8 tls-cert]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-w6w7p" podUID="808d6594-95e7-4cf0-a5d2-918a905f668e"
Oct 30 13:08:45.733082 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:45.733046    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwdk8\" (UniqueName: \"kubernetes.io/projected/808d6594-95e7-4cf0-a5d2-918a905f668e-kube-api-access-kwdk8\") pod \"authorino-6889cbd8c4-w6w7p\" (UID: \"808d6594-95e7-4cf0-a5d2-918a905f668e\") " pod="kuadrant-system/authorino-6889cbd8c4-w6w7p"
Oct 30 13:08:45.733298 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:45.733104    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/808d6594-95e7-4cf0-a5d2-918a905f668e-tls-cert\") pod \"authorino-6889cbd8c4-w6w7p\" (UID: \"808d6594-95e7-4cf0-a5d2-918a905f668e\") " pod="kuadrant-system/authorino-6889cbd8c4-w6w7p"
Oct 30 13:08:45.834336 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:45.834290    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwdk8\" (UniqueName: \"kubernetes.io/projected/808d6594-95e7-4cf0-a5d2-918a905f668e-kube-api-access-kwdk8\") pod \"authorino-6889cbd8c4-w6w7p\" (UID: \"808d6594-95e7-4cf0-a5d2-918a905f668e\") " pod="kuadrant-system/authorino-6889cbd8c4-w6w7p"
Oct 30 13:08:45.834514 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:45.834371    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/808d6594-95e7-4cf0-a5d2-918a905f668e-tls-cert\") pod \"authorino-6889cbd8c4-w6w7p\" (UID: \"808d6594-95e7-4cf0-a5d2-918a905f668e\") " pod="kuadrant-system/authorino-6889cbd8c4-w6w7p"
Oct 30 13:08:45.836884 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:45.836855    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/808d6594-95e7-4cf0-a5d2-918a905f668e-tls-cert\") pod \"authorino-6889cbd8c4-w6w7p\" (UID: \"808d6594-95e7-4cf0-a5d2-918a905f668e\") " pod="kuadrant-system/authorino-6889cbd8c4-w6w7p"
Oct 30 13:08:45.854485 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:45.854446    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwdk8\" (UniqueName: \"kubernetes.io/projected/808d6594-95e7-4cf0-a5d2-918a905f668e-kube-api-access-kwdk8\") pod \"authorino-6889cbd8c4-w6w7p\" (UID: \"808d6594-95e7-4cf0-a5d2-918a905f668e\") " pod="kuadrant-system/authorino-6889cbd8c4-w6w7p"
Oct 30 13:08:45.903777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:45.903745    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-n6nk4"]
Oct 30 13:08:45.905708 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:08:45.905678    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode04e348c_99c2_419c_8213_365c6607d27f.slice/crio-e3e6a2ca5ff8b89ac6fa4ecd5c85f96bf1632cd160c69e783a08b8502d478c62 WatchSource:0}: Error finding container e3e6a2ca5ff8b89ac6fa4ecd5c85f96bf1632cd160c69e783a08b8502d478c62: Status 404 returned error can't find the container with id e3e6a2ca5ff8b89ac6fa4ecd5c85f96bf1632cd160c69e783a08b8502d478c62
Oct 30 13:08:45.926903 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:45.926867    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-n6nk4" event={"ID":"e04e348c-99c2-419c-8213-365c6607d27f","Type":"ContainerStarted","Data":"e3e6a2ca5ff8b89ac6fa4ecd5c85f96bf1632cd160c69e783a08b8502d478c62"}
Oct 30 13:08:45.927022 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:45.926928    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-w6w7p"
Oct 30 13:08:45.935034 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:45.935009    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-w6w7p"
Oct 30 13:08:46.035543 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:46.035504    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/808d6594-95e7-4cf0-a5d2-918a905f668e-tls-cert\" (UniqueName: \"kubernetes.io/secret/808d6594-95e7-4cf0-a5d2-918a905f668e-tls-cert\") pod \"808d6594-95e7-4cf0-a5d2-918a905f668e\" (UID: \"808d6594-95e7-4cf0-a5d2-918a905f668e\") "
Oct 30 13:08:46.035735 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:46.035621    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/808d6594-95e7-4cf0-a5d2-918a905f668e-kube-api-access-kwdk8\" (UniqueName: \"kubernetes.io/projected/808d6594-95e7-4cf0-a5d2-918a905f668e-kube-api-access-kwdk8\") pod \"808d6594-95e7-4cf0-a5d2-918a905f668e\" (UID: \"808d6594-95e7-4cf0-a5d2-918a905f668e\") "
Oct 30 13:08:46.037756 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:46.037730    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/808d6594-95e7-4cf0-a5d2-918a905f668e-tls-cert" pod "808d6594-95e7-4cf0-a5d2-918a905f668e" (UID: "808d6594-95e7-4cf0-a5d2-918a905f668e"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:08:46.038344 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:46.038318    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/808d6594-95e7-4cf0-a5d2-918a905f668e-kube-api-access-kwdk8" pod "808d6594-95e7-4cf0-a5d2-918a905f668e" (UID: "808d6594-95e7-4cf0-a5d2-918a905f668e"). InnerVolumeSpecName "kube-api-access-kwdk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:08:46.136986 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:46.136949    5093 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/808d6594-95e7-4cf0-a5d2-918a905f668e-tls-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:08:46.136986 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:46.136983    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwdk8\" (UniqueName: \"kubernetes.io/projected/808d6594-95e7-4cf0-a5d2-918a905f668e-kube-api-access-kwdk8\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:08:46.442277 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:46.442235    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-knngs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": read tcp 10.132.0.2:42386->10.132.0.33:15021: read: connection reset by peer" start-of-body=
Oct 30 13:08:46.442684 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:46.442298    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs" podUID="c417d75a-c5bf-42fa-8aff-335a63c1295e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": read tcp 10.132.0.2:42386->10.132.0.33:15021: read: connection reset by peer"
Oct 30 13:08:46.934878 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:46.934571    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-n6nk4" event={"ID":"e04e348c-99c2-419c-8213-365c6607d27f","Type":"ContainerStarted","Data":"e45bdf041417bdab1fd724aece31a98f322125a90d85ba21ffe3004f575d60e2"}
Oct 30 13:08:46.934878 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:46.934675    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-n6nk4" podUID="e04e348c-99c2-419c-8213-365c6607d27f" containerName="authorino" containerID="cri-o://e45bdf041417bdab1fd724aece31a98f322125a90d85ba21ffe3004f575d60e2" gracePeriod=30
Oct 30 13:08:46.936537 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:46.936509    5093 generic.go:356] "Generic (PLEG): container finished" podID="c417d75a-c5bf-42fa-8aff-335a63c1295e" containerID="ebce1b79b819e2495a9dc6584f37315e5b653389f7a936ae939e2c46153e4daf" exitCode=0
Oct 30 13:08:46.936645 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:46.936591    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-w6w7p"
Oct 30 13:08:46.936645 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:46.936588    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs" event={"ID":"c417d75a-c5bf-42fa-8aff-335a63c1295e","Type":"ContainerDied","Data":"ebce1b79b819e2495a9dc6584f37315e5b653389f7a936ae939e2c46153e4daf"}
Oct 30 13:08:46.937092 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:46.937072    5093 scope.go:122] "RemoveContainer" containerID="ebce1b79b819e2495a9dc6584f37315e5b653389f7a936ae939e2c46153e4daf"
Oct 30 13:08:46.938360 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:46.938322    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:08:46.938445 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:46.938422    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:08:46.938514 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:46.938497    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:08:46.979394 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:46.979338    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-n6nk4" podStartSLOduration=1.581371364 podStartE2EDuration="1.979319868s" totalImagesPullingTime="397.948504ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:46.977829348 +0000 UTC m=+346.125096601" watchObservedRunningTime="2026-10-30 13:08:46.979319868 +0000 UTC m=+346.126587110"
Oct 30 13:08:47.128277 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:47.122514    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-w6w7p"]
Oct 30 13:08:47.131101 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:47.130221    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-w6w7p"]
Oct 30 13:08:47.215086 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:47.215060    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-n6nk4"
Oct 30 13:08:47.248666 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:47.248636    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e04e348c-99c2-419c-8213-365c6607d27f-kube-api-access-zsx5h\" (UniqueName: \"kubernetes.io/projected/e04e348c-99c2-419c-8213-365c6607d27f-kube-api-access-zsx5h\") pod \"e04e348c-99c2-419c-8213-365c6607d27f\" (UID: \"e04e348c-99c2-419c-8213-365c6607d27f\") "
Oct 30 13:08:47.250706 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:47.250673    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e04e348c-99c2-419c-8213-365c6607d27f-kube-api-access-zsx5h" pod "e04e348c-99c2-419c-8213-365c6607d27f" (UID: "e04e348c-99c2-419c-8213-365c6607d27f"). InnerVolumeSpecName "kube-api-access-zsx5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:08:47.350215 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:47.350169    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsx5h\" (UniqueName: \"kubernetes.io/projected/e04e348c-99c2-419c-8213-365c6607d27f-kube-api-access-zsx5h\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:08:47.957691 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:47.957652    5093 generic.go:356] "Generic (PLEG): container finished" podID="e04e348c-99c2-419c-8213-365c6607d27f" containerID="e45bdf041417bdab1fd724aece31a98f322125a90d85ba21ffe3004f575d60e2" exitCode=0
Oct 30 13:08:47.958172 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:47.957703    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-n6nk4"
Oct 30 13:08:47.958172 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:47.957736    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-n6nk4" event={"ID":"e04e348c-99c2-419c-8213-365c6607d27f","Type":"ContainerDied","Data":"e45bdf041417bdab1fd724aece31a98f322125a90d85ba21ffe3004f575d60e2"}
Oct 30 13:08:47.958172 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:47.957783    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-n6nk4" event={"ID":"e04e348c-99c2-419c-8213-365c6607d27f","Type":"ContainerDied","Data":"e3e6a2ca5ff8b89ac6fa4ecd5c85f96bf1632cd160c69e783a08b8502d478c62"}
Oct 30 13:08:47.958172 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:47.957806    5093 scope.go:122] "RemoveContainer" containerID="e45bdf041417bdab1fd724aece31a98f322125a90d85ba21ffe3004f575d60e2"
Oct 30 13:08:47.959982 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:47.959953    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs" event={"ID":"c417d75a-c5bf-42fa-8aff-335a63c1295e","Type":"ContainerStarted","Data":"1b5bad4cf12c2a37eacf7e5769c7faad6086bc4e3db76a8e98573fb316c3d9e9"}
Oct 30 13:08:47.967733 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:47.967714    5093 scope.go:122] "RemoveContainer" containerID="e45bdf041417bdab1fd724aece31a98f322125a90d85ba21ffe3004f575d60e2"
Oct 30 13:08:47.968021 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:08:47.968002    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e45bdf041417bdab1fd724aece31a98f322125a90d85ba21ffe3004f575d60e2\": container with ID starting with e45bdf041417bdab1fd724aece31a98f322125a90d85ba21ffe3004f575d60e2 not found: ID does not exist" containerID="e45bdf041417bdab1fd724aece31a98f322125a90d85ba21ffe3004f575d60e2"
Oct 30 13:08:47.968067 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:47.968029    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e45bdf041417bdab1fd724aece31a98f322125a90d85ba21ffe3004f575d60e2"} err="failed to get container status \"e45bdf041417bdab1fd724aece31a98f322125a90d85ba21ffe3004f575d60e2\": rpc error: code = NotFound desc = could not find container \"e45bdf041417bdab1fd724aece31a98f322125a90d85ba21ffe3004f575d60e2\": container with ID starting with e45bdf041417bdab1fd724aece31a98f322125a90d85ba21ffe3004f575d60e2 not found: ID does not exist"
Oct 30 13:08:48.065531 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:48.065495    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-n6nk4"]
Oct 30 13:08:48.067342 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:48.067316    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-n6nk4"]
Oct 30 13:08:48.644157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:48.634555    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs"
Oct 30 13:08:48.644157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:48.634689    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-knngs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Oct 30 13:08:48.644157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:48.634735    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs" podUID="c417d75a-c5bf-42fa-8aff-335a63c1295e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Oct 30 13:08:48.997160 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:48.997110    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="808d6594-95e7-4cf0-a5d2-918a905f668e" path="/var/lib/kubelet/pods/808d6594-95e7-4cf0-a5d2-918a905f668e/volumes"
Oct 30 13:08:48.997656 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:48.997384    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e04e348c-99c2-419c-8213-365c6607d27f" path="/var/lib/kubelet/pods/e04e348c-99c2-419c-8213-365c6607d27f/volumes"
Oct 30 13:08:49.641640 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:49.641595    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-knngs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Oct 30 13:08:49.641971 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:49.641938    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs" podUID="c417d75a-c5bf-42fa-8aff-335a63c1295e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Oct 30 13:08:50.646626 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:50.646583    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-knngs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Oct 30 13:08:50.647278 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:50.647244    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs" podUID="c417d75a-c5bf-42fa-8aff-335a63c1295e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Oct 30 13:08:51.179200 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:08:51.179157    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:08:51.647218 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:51.642587    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-knngs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Oct 30 13:08:51.647218 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:51.642674    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs" podUID="c417d75a-c5bf-42fa-8aff-335a63c1295e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Oct 30 13:08:52.636218 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:52.635192    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-knngs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Oct 30 13:08:52.636218 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:52.635271    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs" podUID="c417d75a-c5bf-42fa-8aff-335a63c1295e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Oct 30 13:08:53.637697 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:53.637667    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs"
Oct 30 13:08:53.638097 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:53.637949    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs"
Oct 30 13:08:53.638601 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:08:53.638585    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs"
Oct 30 13:09:21.165788 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:09:21.165752    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:09:51.166040 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:09:51.165946    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:10:21.165879 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:10:21.165840    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:10:51.167682 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:10:51.167645    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:10:54.821546 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:10:54.821491    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-z5nd6"]
Oct 30 13:10:54.822996 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:10:54.822966    5093 cpu_manager.go:414] "containerMap: removing container" podUID="e04e348c-99c2-419c-8213-365c6607d27f" containerName="authorino"
Oct 30 13:10:54.822996 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:10:54.822997    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e04e348c-99c2-419c-8213-365c6607d27f" containerName="authorino"
Oct 30 13:10:54.823208 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:10:54.823125    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="e04e348c-99c2-419c-8213-365c6607d27f" containerName="authorino"
Oct 30 13:10:54.826606 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:10:54.826586    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z5nd6"
Oct 30 13:10:54.841655 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:10:54.841632    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-z5nd6"]
Oct 30 13:10:54.852577 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:10:54.852552    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 13:10:54.852882 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:10:54.852847    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Oct 30 13:10:54.853062 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:10:54.852877    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 13:10:54.855797 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:10:54.855775    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-xkkzc\""
Oct 30 13:10:54.864195 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:10:54.864169    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47d931a7-02ab-412b-a753-6bead66a76cd-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z5nd6\" (UID: \"47d931a7-02ab-412b-a753-6bead66a76cd\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z5nd6"
Oct 30 13:10:54.864297 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:10:54.864216    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47d931a7-02ab-412b-a753-6bead66a76cd-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z5nd6\" (UID: \"47d931a7-02ab-412b-a753-6bead66a76cd\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z5nd6"
Oct 30 13:10:54.864297 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:10:54.864239    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47d931a7-02ab-412b-a753-6bead66a76cd-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z5nd6\" (UID: \"47d931a7-02ab-412b-a753-6bead66a76cd\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z5nd6"
Oct 30 13:10:54.864397 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:10:54.864378    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47d931a7-02ab-412b-a753-6bead66a76cd-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z5nd6\" (UID: \"47d931a7-02ab-412b-a753-6bead66a76cd\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z5nd6"
Oct 30 13:10:54.864446 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:10:54.864427    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5ml7\" (UniqueName: \"kubernetes.io/projected/47d931a7-02ab-412b-a753-6bead66a76cd-kube-api-access-j5ml7\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z5nd6\" (UID: \"47d931a7-02ab-412b-a753-6bead66a76cd\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z5nd6"
Oct 30 13:10:54.864525 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:10:54.864509    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47d931a7-02ab-412b-a753-6bead66a76cd-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z5nd6\" (UID: \"47d931a7-02ab-412b-a753-6bead66a76cd\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z5nd6"
Oct 30 13:10:54.965731 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:10:54.965696    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47d931a7-02ab-412b-a753-6bead66a76cd-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z5nd6\" (UID: \"47d931a7-02ab-412b-a753-6bead66a76cd\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z5nd6"
Oct 30 13:10:54.965731 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:10:54.965736    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47d931a7-02ab-412b-a753-6bead66a76cd-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z5nd6\" (UID: \"47d931a7-02ab-412b-a753-6bead66a76cd\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z5nd6"
Oct 30 13:10:54.966020 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:10:54.965762    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47d931a7-02ab-412b-a753-6bead66a76cd-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z5nd6\" (UID: \"47d931a7-02ab-412b-a753-6bead66a76cd\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z5nd6"
Oct 30 13:10:54.966020 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:10:54.965891    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5ml7\" (UniqueName: \"kubernetes.io/projected/47d931a7-02ab-412b-a753-6bead66a76cd-kube-api-access-j5ml7\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z5nd6\" (UID: \"47d931a7-02ab-412b-a753-6bead66a76cd\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z5nd6"
Oct 30 13:10:54.966020 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:10:54.965938    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47d931a7-02ab-412b-a753-6bead66a76cd-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z5nd6\" (UID: \"47d931a7-02ab-412b-a753-6bead66a76cd\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z5nd6"
Oct 30 13:10:54.966020 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:10:54.965994    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47d931a7-02ab-412b-a753-6bead66a76cd-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z5nd6\" (UID: \"47d931a7-02ab-412b-a753-6bead66a76cd\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z5nd6"
Oct 30 13:10:54.966264 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:10:54.966241    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47d931a7-02ab-412b-a753-6bead66a76cd-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z5nd6\" (UID: \"47d931a7-02ab-412b-a753-6bead66a76cd\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z5nd6"
Oct 30 13:10:54.966313 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:10:54.966272    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47d931a7-02ab-412b-a753-6bead66a76cd-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z5nd6\" (UID: \"47d931a7-02ab-412b-a753-6bead66a76cd\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z5nd6"
Oct 30 13:10:54.966358 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:10:54.966336    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47d931a7-02ab-412b-a753-6bead66a76cd-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z5nd6\" (UID: \"47d931a7-02ab-412b-a753-6bead66a76cd\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z5nd6"
Oct 30 13:10:54.968094 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:10:54.968073    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47d931a7-02ab-412b-a753-6bead66a76cd-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z5nd6\" (UID: \"47d931a7-02ab-412b-a753-6bead66a76cd\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z5nd6"
Oct 30 13:10:54.968348 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:10:54.968330    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47d931a7-02ab-412b-a753-6bead66a76cd-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z5nd6\" (UID: \"47d931a7-02ab-412b-a753-6bead66a76cd\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z5nd6"
Oct 30 13:10:54.986562 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:10:54.986535    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5ml7\" (UniqueName: \"kubernetes.io/projected/47d931a7-02ab-412b-a753-6bead66a76cd-kube-api-access-j5ml7\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z5nd6\" (UID: \"47d931a7-02ab-412b-a753-6bead66a76cd\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z5nd6"
Oct 30 13:10:55.141400 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:10:55.141308    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z5nd6"
Oct 30 13:10:55.452725 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:10:55.452680    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-z5nd6"]
Oct 30 13:10:55.457855 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:10:55.457823    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47d931a7_02ab_412b_a753_6bead66a76cd.slice/crio-dae678759a448516f43ce88ea94f14f953dedf800e4bcf7d3b5c9cbe6912f782 WatchSource:0}: Error finding container dae678759a448516f43ce88ea94f14f953dedf800e4bcf7d3b5c9cbe6912f782: Status 404 returned error can't find the container with id dae678759a448516f43ce88ea94f14f953dedf800e4bcf7d3b5c9cbe6912f782
Oct 30 13:10:55.626704 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:10:55.626660    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z5nd6" event={"ID":"47d931a7-02ab-412b-a753-6bead66a76cd","Type":"ContainerStarted","Data":"dae678759a448516f43ce88ea94f14f953dedf800e4bcf7d3b5c9cbe6912f782"}
Oct 30 13:10:56.634665 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:10:56.634629    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z5nd6" event={"ID":"47d931a7-02ab-412b-a753-6bead66a76cd","Type":"ContainerStarted","Data":"9939270b7974f165851fffc0368af6daea7ee0f1ffe4e74ed6e1f04a9e0d7450"}
Oct 30 13:10:56.681778 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:10:56.681725    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z5nd6" podStartSLOduration=1.62231569 podStartE2EDuration="2.681712289s" totalImagesPullingTime="1.059396599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:56.681413604 +0000 UTC m=+475.828680847" watchObservedRunningTime="2026-10-30 13:10:56.681712289 +0000 UTC m=+475.828979557"
Oct 30 13:10:57.141874 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:10:57.141834    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z5nd6"
Oct 30 13:10:59.956732 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:10:59.956697    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b867hfz"]
Oct 30 13:10:59.960500 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:10:59.960477    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b867hfz"
Oct 30 13:10:59.978066 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:10:59.978037    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b867hfz"]
Oct 30 13:10:59.988798 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:10:59.988771    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Oct 30 13:11:00.016900 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:00.016864    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1007aa25-0492-47a8-b0d5-a864fec1022a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b867hfz\" (UID: \"1007aa25-0492-47a8-b0d5-a864fec1022a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b867hfz"
Oct 30 13:11:00.016900 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:00.016905    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1007aa25-0492-47a8-b0d5-a864fec1022a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b867hfz\" (UID: \"1007aa25-0492-47a8-b0d5-a864fec1022a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b867hfz"
Oct 30 13:11:00.017123 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:00.016938    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1007aa25-0492-47a8-b0d5-a864fec1022a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b867hfz\" (UID: \"1007aa25-0492-47a8-b0d5-a864fec1022a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b867hfz"
Oct 30 13:11:00.017123 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:00.017065    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1007aa25-0492-47a8-b0d5-a864fec1022a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b867hfz\" (UID: \"1007aa25-0492-47a8-b0d5-a864fec1022a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b867hfz"
Oct 30 13:11:00.017123 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:00.017093    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1007aa25-0492-47a8-b0d5-a864fec1022a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b867hfz\" (UID: \"1007aa25-0492-47a8-b0d5-a864fec1022a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b867hfz"
Oct 30 13:11:00.017123 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:00.017112    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnz95\" (UniqueName: \"kubernetes.io/projected/1007aa25-0492-47a8-b0d5-a864fec1022a-kube-api-access-dnz95\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b867hfz\" (UID: \"1007aa25-0492-47a8-b0d5-a864fec1022a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b867hfz"
Oct 30 13:11:00.118354 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:00.118300    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1007aa25-0492-47a8-b0d5-a864fec1022a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b867hfz\" (UID: \"1007aa25-0492-47a8-b0d5-a864fec1022a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b867hfz"
Oct 30 13:11:00.118354 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:00.118362    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1007aa25-0492-47a8-b0d5-a864fec1022a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b867hfz\" (UID: \"1007aa25-0492-47a8-b0d5-a864fec1022a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b867hfz"
Oct 30 13:11:00.118624 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:00.118425    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1007aa25-0492-47a8-b0d5-a864fec1022a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b867hfz\" (UID: \"1007aa25-0492-47a8-b0d5-a864fec1022a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b867hfz"
Oct 30 13:11:00.118710 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:00.118683    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1007aa25-0492-47a8-b0d5-a864fec1022a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b867hfz\" (UID: \"1007aa25-0492-47a8-b0d5-a864fec1022a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b867hfz"
Oct 30 13:11:00.118776 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:00.118726    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1007aa25-0492-47a8-b0d5-a864fec1022a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b867hfz\" (UID: \"1007aa25-0492-47a8-b0d5-a864fec1022a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b867hfz"
Oct 30 13:11:00.118776 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:00.118754    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnz95\" (UniqueName: \"kubernetes.io/projected/1007aa25-0492-47a8-b0d5-a864fec1022a-kube-api-access-dnz95\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b867hfz\" (UID: \"1007aa25-0492-47a8-b0d5-a864fec1022a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b867hfz"
Oct 30 13:11:00.118876 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:00.118800    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1007aa25-0492-47a8-b0d5-a864fec1022a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b867hfz\" (UID: \"1007aa25-0492-47a8-b0d5-a864fec1022a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b867hfz"
Oct 30 13:11:00.118876 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:00.118820    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1007aa25-0492-47a8-b0d5-a864fec1022a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b867hfz\" (UID: \"1007aa25-0492-47a8-b0d5-a864fec1022a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b867hfz"
Oct 30 13:11:00.119166 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:00.119115    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1007aa25-0492-47a8-b0d5-a864fec1022a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b867hfz\" (UID: \"1007aa25-0492-47a8-b0d5-a864fec1022a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b867hfz"
Oct 30 13:11:00.120851 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:00.120824    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1007aa25-0492-47a8-b0d5-a864fec1022a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b867hfz\" (UID: \"1007aa25-0492-47a8-b0d5-a864fec1022a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b867hfz"
Oct 30 13:11:00.121232 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:00.121215    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1007aa25-0492-47a8-b0d5-a864fec1022a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b867hfz\" (UID: \"1007aa25-0492-47a8-b0d5-a864fec1022a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b867hfz"
Oct 30 13:11:00.146918 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:00.146890    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnz95\" (UniqueName: \"kubernetes.io/projected/1007aa25-0492-47a8-b0d5-a864fec1022a-kube-api-access-dnz95\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b867hfz\" (UID: \"1007aa25-0492-47a8-b0d5-a864fec1022a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b867hfz"
Oct 30 13:11:00.274822 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:00.274723    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b867hfz"
Oct 30 13:11:00.587454 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:00.587419    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b867hfz"]
Oct 30 13:11:00.588968 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:11:00.588934    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1007aa25_0492_47a8_b0d5_a864fec1022a.slice/crio-6423e5d3f072c010bb66fe2eb050bef4128ede5126bd581194c2c775201b14e4 WatchSource:0}: Error finding container 6423e5d3f072c010bb66fe2eb050bef4128ede5126bd581194c2c775201b14e4: Status 404 returned error can't find the container with id 6423e5d3f072c010bb66fe2eb050bef4128ede5126bd581194c2c775201b14e4
Oct 30 13:11:00.659799 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:00.659757    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b867hfz" event={"ID":"1007aa25-0492-47a8-b0d5-a864fec1022a","Type":"ContainerStarted","Data":"6423e5d3f072c010bb66fe2eb050bef4128ede5126bd581194c2c775201b14e4"}
Oct 30 13:11:01.666984 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:01.666943    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b867hfz" event={"ID":"1007aa25-0492-47a8-b0d5-a864fec1022a","Type":"ContainerStarted","Data":"780196d4742547da702d643f5c6b33ab04eec382d06422c18c1e8082064699b1"}
Oct 30 13:11:01.708950 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:01.708895    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b867hfz" podStartSLOduration=2.5175614299999998 podStartE2EDuration="2.70887989s" totalImagesPullingTime="191.31846ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:01.707957153 +0000 UTC m=+480.855224394" watchObservedRunningTime="2026-10-30 13:11:01.70887989 +0000 UTC m=+480.856147131"
Oct 30 13:11:02.275318 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:02.275281    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b867hfz"
Oct 30 13:11:04.543020 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:04.542983    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7hplr"]
Oct 30 13:11:04.546727 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:04.546695    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7hplr"
Oct 30 13:11:04.578459 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:04.578429    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Oct 30 13:11:04.582178 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:04.582145    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7hplr"]
Oct 30 13:11:04.663515 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:04.663477    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51cafd88-c303-4185-ac30-aa9f20b1ed4f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7hplr\" (UID: \"51cafd88-c303-4185-ac30-aa9f20b1ed4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7hplr"
Oct 30 13:11:04.663687 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:04.663532    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56ppl\" (UniqueName: \"kubernetes.io/projected/51cafd88-c303-4185-ac30-aa9f20b1ed4f-kube-api-access-56ppl\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7hplr\" (UID: \"51cafd88-c303-4185-ac30-aa9f20b1ed4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7hplr"
Oct 30 13:11:04.663687 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:04.663638    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51cafd88-c303-4185-ac30-aa9f20b1ed4f-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7hplr\" (UID: \"51cafd88-c303-4185-ac30-aa9f20b1ed4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7hplr"
Oct 30 13:11:04.663687 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:04.663675    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51cafd88-c303-4185-ac30-aa9f20b1ed4f-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7hplr\" (UID: \"51cafd88-c303-4185-ac30-aa9f20b1ed4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7hplr"
Oct 30 13:11:04.663793 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:04.663695    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51cafd88-c303-4185-ac30-aa9f20b1ed4f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7hplr\" (UID: \"51cafd88-c303-4185-ac30-aa9f20b1ed4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7hplr"
Oct 30 13:11:04.663827 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:04.663786    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51cafd88-c303-4185-ac30-aa9f20b1ed4f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7hplr\" (UID: \"51cafd88-c303-4185-ac30-aa9f20b1ed4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7hplr"
Oct 30 13:11:04.764549 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:04.764509    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56ppl\" (UniqueName: \"kubernetes.io/projected/51cafd88-c303-4185-ac30-aa9f20b1ed4f-kube-api-access-56ppl\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7hplr\" (UID: \"51cafd88-c303-4185-ac30-aa9f20b1ed4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7hplr"
Oct 30 13:11:04.764744 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:04.764563    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51cafd88-c303-4185-ac30-aa9f20b1ed4f-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7hplr\" (UID: \"51cafd88-c303-4185-ac30-aa9f20b1ed4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7hplr"
Oct 30 13:11:04.764744 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:04.764674    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51cafd88-c303-4185-ac30-aa9f20b1ed4f-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7hplr\" (UID: \"51cafd88-c303-4185-ac30-aa9f20b1ed4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7hplr"
Oct 30 13:11:04.764744 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:04.764713    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51cafd88-c303-4185-ac30-aa9f20b1ed4f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7hplr\" (UID: \"51cafd88-c303-4185-ac30-aa9f20b1ed4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7hplr"
Oct 30 13:11:04.764896 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:04.764849    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51cafd88-c303-4185-ac30-aa9f20b1ed4f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7hplr\" (UID: \"51cafd88-c303-4185-ac30-aa9f20b1ed4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7hplr"
Oct 30 13:11:04.764952 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:04.764932    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51cafd88-c303-4185-ac30-aa9f20b1ed4f-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7hplr\" (UID: \"51cafd88-c303-4185-ac30-aa9f20b1ed4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7hplr"
Oct 30 13:11:04.765005 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:04.764947    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51cafd88-c303-4185-ac30-aa9f20b1ed4f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7hplr\" (UID: \"51cafd88-c303-4185-ac30-aa9f20b1ed4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7hplr"
Oct 30 13:11:04.765005 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:04.764968    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51cafd88-c303-4185-ac30-aa9f20b1ed4f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7hplr\" (UID: \"51cafd88-c303-4185-ac30-aa9f20b1ed4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7hplr"
Oct 30 13:11:04.765241 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:04.765221    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51cafd88-c303-4185-ac30-aa9f20b1ed4f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7hplr\" (UID: \"51cafd88-c303-4185-ac30-aa9f20b1ed4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7hplr"
Oct 30 13:11:04.766968 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:04.766942    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51cafd88-c303-4185-ac30-aa9f20b1ed4f-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7hplr\" (UID: \"51cafd88-c303-4185-ac30-aa9f20b1ed4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7hplr"
Oct 30 13:11:04.767280 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:04.767257    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51cafd88-c303-4185-ac30-aa9f20b1ed4f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7hplr\" (UID: \"51cafd88-c303-4185-ac30-aa9f20b1ed4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7hplr"
Oct 30 13:11:04.786842 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:04.786811    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56ppl\" (UniqueName: \"kubernetes.io/projected/51cafd88-c303-4185-ac30-aa9f20b1ed4f-kube-api-access-56ppl\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7hplr\" (UID: \"51cafd88-c303-4185-ac30-aa9f20b1ed4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7hplr"
Oct 30 13:11:04.860882 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:04.860795    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7hplr"
Oct 30 13:11:05.142246 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:05.142164    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z5nd6"
Oct 30 13:11:05.155108 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:05.155082    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z5nd6"
Oct 30 13:11:05.160540 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:05.160517    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7hplr"]
Oct 30 13:11:05.162341 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:11:05.162306    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51cafd88_c303_4185_ac30_aa9f20b1ed4f.slice/crio-b15a432dff0776d5da35fa9eeab643a5d695447d5eedb5b223a7e3c22e9e3f72 WatchSource:0}: Error finding container b15a432dff0776d5da35fa9eeab643a5d695447d5eedb5b223a7e3c22e9e3f72: Status 404 returned error can't find the container with id b15a432dff0776d5da35fa9eeab643a5d695447d5eedb5b223a7e3c22e9e3f72
Oct 30 13:11:05.694344 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:05.694301    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7hplr" event={"ID":"51cafd88-c303-4185-ac30-aa9f20b1ed4f","Type":"ContainerStarted","Data":"c5e4363f7320466ab4cdaa4c71dd980c90ed6eccfc1e0412d8b29fcb662483df"}
Oct 30 13:11:05.694818 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:05.694379    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7hplr" event={"ID":"51cafd88-c303-4185-ac30-aa9f20b1ed4f","Type":"ContainerStarted","Data":"b15a432dff0776d5da35fa9eeab643a5d695447d5eedb5b223a7e3c22e9e3f72"}
Oct 30 13:11:05.705870 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:05.705840    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z5nd6"
Oct 30 13:11:05.737501 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:05.737447    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7hplr" podStartSLOduration=1.55264485 podStartE2EDuration="1.737432661s" totalImagesPullingTime="184.787811ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:05.735936809 +0000 UTC m=+484.883204063" watchObservedRunningTime="2026-10-30 13:11:05.737432661 +0000 UTC m=+484.884699902"
Oct 30 13:11:05.861383 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:05.861350    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7hplr"
Oct 30 13:11:06.355205 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:06.355162    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg"]
Oct 30 13:11:06.360389 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:06.360360    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg"
Oct 30 13:11:06.378156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:06.377712    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg"]
Oct 30 13:11:06.388141 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:06.388102    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Oct 30 13:11:06.482904 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:06.482859    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea079ac1-f502-4b5f-b16a-c91e2de98de8-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg\" (UID: \"ea079ac1-f502-4b5f-b16a-c91e2de98de8\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg"
Oct 30 13:11:06.483107 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:06.482982    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea079ac1-f502-4b5f-b16a-c91e2de98de8-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg\" (UID: \"ea079ac1-f502-4b5f-b16a-c91e2de98de8\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg"
Oct 30 13:11:06.483107 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:06.483014    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea079ac1-f502-4b5f-b16a-c91e2de98de8-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg\" (UID: \"ea079ac1-f502-4b5f-b16a-c91e2de98de8\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg"
Oct 30 13:11:06.483107 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:06.483071    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j4js\" (UniqueName: \"kubernetes.io/projected/ea079ac1-f502-4b5f-b16a-c91e2de98de8-kube-api-access-5j4js\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg\" (UID: \"ea079ac1-f502-4b5f-b16a-c91e2de98de8\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg"
Oct 30 13:11:06.483317 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:06.483116    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea079ac1-f502-4b5f-b16a-c91e2de98de8-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg\" (UID: \"ea079ac1-f502-4b5f-b16a-c91e2de98de8\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg"
Oct 30 13:11:06.483317 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:06.483147    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea079ac1-f502-4b5f-b16a-c91e2de98de8-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg\" (UID: \"ea079ac1-f502-4b5f-b16a-c91e2de98de8\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg"
Oct 30 13:11:06.584244 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:06.584199    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea079ac1-f502-4b5f-b16a-c91e2de98de8-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg\" (UID: \"ea079ac1-f502-4b5f-b16a-c91e2de98de8\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg"
Oct 30 13:11:06.584426 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:06.584324    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea079ac1-f502-4b5f-b16a-c91e2de98de8-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg\" (UID: \"ea079ac1-f502-4b5f-b16a-c91e2de98de8\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg"
Oct 30 13:11:06.584426 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:06.584354    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea079ac1-f502-4b5f-b16a-c91e2de98de8-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg\" (UID: \"ea079ac1-f502-4b5f-b16a-c91e2de98de8\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg"
Oct 30 13:11:06.584426 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:06.584389    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j4js\" (UniqueName: \"kubernetes.io/projected/ea079ac1-f502-4b5f-b16a-c91e2de98de8-kube-api-access-5j4js\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg\" (UID: \"ea079ac1-f502-4b5f-b16a-c91e2de98de8\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg"
Oct 30 13:11:06.584426 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:06.584421    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea079ac1-f502-4b5f-b16a-c91e2de98de8-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg\" (UID: \"ea079ac1-f502-4b5f-b16a-c91e2de98de8\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg"
Oct 30 13:11:06.584567 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:06.584445    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea079ac1-f502-4b5f-b16a-c91e2de98de8-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg\" (UID: \"ea079ac1-f502-4b5f-b16a-c91e2de98de8\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg"
Oct 30 13:11:06.584905 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:06.584872    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea079ac1-f502-4b5f-b16a-c91e2de98de8-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg\" (UID: \"ea079ac1-f502-4b5f-b16a-c91e2de98de8\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg"
Oct 30 13:11:06.585118 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:06.585069    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea079ac1-f502-4b5f-b16a-c91e2de98de8-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg\" (UID: \"ea079ac1-f502-4b5f-b16a-c91e2de98de8\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg"
Oct 30 13:11:06.585329 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:06.585306    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea079ac1-f502-4b5f-b16a-c91e2de98de8-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg\" (UID: \"ea079ac1-f502-4b5f-b16a-c91e2de98de8\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg"
Oct 30 13:11:06.587330 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:06.587302    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea079ac1-f502-4b5f-b16a-c91e2de98de8-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg\" (UID: \"ea079ac1-f502-4b5f-b16a-c91e2de98de8\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg"
Oct 30 13:11:06.588336 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:06.588311    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea079ac1-f502-4b5f-b16a-c91e2de98de8-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg\" (UID: \"ea079ac1-f502-4b5f-b16a-c91e2de98de8\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg"
Oct 30 13:11:06.605526 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:06.605453    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j4js\" (UniqueName: \"kubernetes.io/projected/ea079ac1-f502-4b5f-b16a-c91e2de98de8-kube-api-access-5j4js\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg\" (UID: \"ea079ac1-f502-4b5f-b16a-c91e2de98de8\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg"
Oct 30 13:11:06.678432 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:06.678391    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg"
Oct 30 13:11:07.021563 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:11:07.021528    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea079ac1_f502_4b5f_b16a_c91e2de98de8.slice/crio-161901e3a015fb5eb0d23db146d42303e9c09a4cd04c46e51ad6a20c527e185f WatchSource:0}: Error finding container 161901e3a015fb5eb0d23db146d42303e9c09a4cd04c46e51ad6a20c527e185f: Status 404 returned error can't find the container with id 161901e3a015fb5eb0d23db146d42303e9c09a4cd04c46e51ad6a20c527e185f
Oct 30 13:11:07.023522 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:07.023490    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg"]
Oct 30 13:11:07.711545 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:07.711507    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg" event={"ID":"ea079ac1-f502-4b5f-b16a-c91e2de98de8","Type":"ContainerStarted","Data":"433e24490316ae5fadbb818f5bf7d4c9b2d468c0414163ab156a7dee736b77ef"}
Oct 30 13:11:07.711756 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:07.711557    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg" event={"ID":"ea079ac1-f502-4b5f-b16a-c91e2de98de8","Type":"ContainerStarted","Data":"161901e3a015fb5eb0d23db146d42303e9c09a4cd04c46e51ad6a20c527e185f"}
Oct 30 13:11:07.759506 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:07.759436    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg" podStartSLOduration=1.556607904 podStartE2EDuration="1.759405943s" totalImagesPullingTime="202.798039ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:07.757487891 +0000 UTC m=+486.904755134" watchObservedRunningTime="2026-10-30 13:11:07.759405943 +0000 UTC m=+486.906673185"
Oct 30 13:11:08.679143 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:08.679087    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg"
Oct 30 13:11:09.242888 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:09.242853    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h"]
Oct 30 13:11:09.246652 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:09.246632    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h"
Oct 30 13:11:09.283702 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:09.283668    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h"]
Oct 30 13:11:09.412165 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:09.412104    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-5rw2h\" (UID: \"35df4dfa-f04b-47b8-86cc-8d7758fcec5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h"
Oct 30 13:11:09.412368 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:09.412217    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-5rw2h\" (UID: \"35df4dfa-f04b-47b8-86cc-8d7758fcec5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h"
Oct 30 13:11:09.412368 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:09.412335    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-5rw2h\" (UID: \"35df4dfa-f04b-47b8-86cc-8d7758fcec5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h"
Oct 30 13:11:09.412479 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:09.412365    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-5rw2h\" (UID: \"35df4dfa-f04b-47b8-86cc-8d7758fcec5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h"
Oct 30 13:11:09.412479 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:09.412471    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-5rw2h\" (UID: \"35df4dfa-f04b-47b8-86cc-8d7758fcec5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h"
Oct 30 13:11:09.412589 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:09.412537    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-5rw2h\" (UID: \"35df4dfa-f04b-47b8-86cc-8d7758fcec5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h"
Oct 30 13:11:09.412643 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:09.412598    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqrhl\" (UniqueName: \"kubernetes.io/projected/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-kube-api-access-zqrhl\") pod \"maas-default-gateway-openshift-default-67bf986f99-5rw2h\" (UID: \"35df4dfa-f04b-47b8-86cc-8d7758fcec5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h"
Oct 30 13:11:09.412902 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:09.412734    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-5rw2h\" (UID: \"35df4dfa-f04b-47b8-86cc-8d7758fcec5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h"
Oct 30 13:11:09.412902 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:09.412825    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-5rw2h\" (UID: \"35df4dfa-f04b-47b8-86cc-8d7758fcec5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h"
Oct 30 13:11:09.514336 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:09.514244    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-5rw2h\" (UID: \"35df4dfa-f04b-47b8-86cc-8d7758fcec5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h"
Oct 30 13:11:09.514336 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:09.514281    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-5rw2h\" (UID: \"35df4dfa-f04b-47b8-86cc-8d7758fcec5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h"
Oct 30 13:11:09.514336 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:09.514312    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-5rw2h\" (UID: \"35df4dfa-f04b-47b8-86cc-8d7758fcec5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h"
Oct 30 13:11:09.514336 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:09.514330    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-5rw2h\" (UID: \"35df4dfa-f04b-47b8-86cc-8d7758fcec5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h"
Oct 30 13:11:09.514686 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:09.514361    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqrhl\" (UniqueName: \"kubernetes.io/projected/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-kube-api-access-zqrhl\") pod \"maas-default-gateway-openshift-default-67bf986f99-5rw2h\" (UID: \"35df4dfa-f04b-47b8-86cc-8d7758fcec5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h"
Oct 30 13:11:09.514686 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:09.514412    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-5rw2h\" (UID: \"35df4dfa-f04b-47b8-86cc-8d7758fcec5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h"
Oct 30 13:11:09.514686 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:09.514438    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-5rw2h\" (UID: \"35df4dfa-f04b-47b8-86cc-8d7758fcec5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h"
Oct 30 13:11:09.514686 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:09.514478    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-5rw2h\" (UID: \"35df4dfa-f04b-47b8-86cc-8d7758fcec5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h"
Oct 30 13:11:09.514686 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:09.514516    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-5rw2h\" (UID: \"35df4dfa-f04b-47b8-86cc-8d7758fcec5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h"
Oct 30 13:11:09.514928 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:09.514806    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-5rw2h\" (UID: \"35df4dfa-f04b-47b8-86cc-8d7758fcec5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h"
Oct 30 13:11:09.514928 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:09.514842    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-5rw2h\" (UID: \"35df4dfa-f04b-47b8-86cc-8d7758fcec5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h"
Oct 30 13:11:09.514928 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:09.514901    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-5rw2h\" (UID: \"35df4dfa-f04b-47b8-86cc-8d7758fcec5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h"
Oct 30 13:11:09.515028 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:09.514937    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-5rw2h\" (UID: \"35df4dfa-f04b-47b8-86cc-8d7758fcec5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h"
Oct 30 13:11:09.515403 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:09.515372    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-5rw2h\" (UID: \"35df4dfa-f04b-47b8-86cc-8d7758fcec5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h"
Oct 30 13:11:09.516946 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:09.516913    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-5rw2h\" (UID: \"35df4dfa-f04b-47b8-86cc-8d7758fcec5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h"
Oct 30 13:11:09.516946 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:09.516934    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-5rw2h\" (UID: \"35df4dfa-f04b-47b8-86cc-8d7758fcec5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h"
Oct 30 13:11:09.534709 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:09.534683    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqrhl\" (UniqueName: \"kubernetes.io/projected/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-kube-api-access-zqrhl\") pod \"maas-default-gateway-openshift-default-67bf986f99-5rw2h\" (UID: \"35df4dfa-f04b-47b8-86cc-8d7758fcec5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h"
Oct 30 13:11:09.534709 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:09.534693    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-5rw2h\" (UID: \"35df4dfa-f04b-47b8-86cc-8d7758fcec5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h"
Oct 30 13:11:09.559632 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:09.559605    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h"
Oct 30 13:11:09.873345 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:09.873316    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h"]
Oct 30 13:11:09.875330 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:11:09.875298    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35df4dfa_f04b_47b8_86cc_8d7758fcec5a.slice/crio-c880e4c888d544dd457fe60c1f4310e304613d0cc62f834f9c1cef372b60574e WatchSource:0}: Error finding container c880e4c888d544dd457fe60c1f4310e304613d0cc62f834f9c1cef372b60574e: Status 404 returned error can't find the container with id c880e4c888d544dd457fe60c1f4310e304613d0cc62f834f9c1cef372b60574e
Oct 30 13:11:09.877608 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:09.877572    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:11:09.877683 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:09.877662    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:11:09.877723 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:09.877706    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:11:10.275577 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:10.275540    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b867hfz"
Oct 30 13:11:10.288614 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:10.288583    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b867hfz"
Oct 30 13:11:10.743164 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:10.741812    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h" event={"ID":"35df4dfa-f04b-47b8-86cc-8d7758fcec5a","Type":"ContainerStarted","Data":"23426448ba7bfae7d45b30d3947bddeb08864abdd57ca661250ab9d2179907d5"}
Oct 30 13:11:10.743164 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:10.741893    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h" event={"ID":"35df4dfa-f04b-47b8-86cc-8d7758fcec5a","Type":"ContainerStarted","Data":"c880e4c888d544dd457fe60c1f4310e304613d0cc62f834f9c1cef372b60574e"}
Oct 30 13:11:10.762004 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:10.761969    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b867hfz"
Oct 30 13:11:10.786353 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:10.786300    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h" podStartSLOduration=1.786284786 podStartE2EDuration="1.786284786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:10.785521723 +0000 UTC m=+489.932788966" watchObservedRunningTime="2026-10-30 13:11:10.786284786 +0000 UTC m=+489.933552027"
Oct 30 13:11:11.561287 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:11.561245    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h"
Oct 30 13:11:11.561984 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:11.561931    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-5rw2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 30 13:11:11.562100 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:11.562004    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h" podUID="35df4dfa-f04b-47b8-86cc-8d7758fcec5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 30 13:11:12.560626 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:12.560583    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-5rw2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 30 13:11:12.560892 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:12.560862    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h" podUID="35df4dfa-f04b-47b8-86cc-8d7758fcec5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 30 13:11:13.560446 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:13.560402    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-5rw2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 30 13:11:13.560860 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:13.560462    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h" podUID="35df4dfa-f04b-47b8-86cc-8d7758fcec5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 30 13:11:14.560895 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:14.560853    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-5rw2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 30 13:11:14.561330 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:14.560923    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h" podUID="35df4dfa-f04b-47b8-86cc-8d7758fcec5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 30 13:11:14.861940 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:14.861846    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7hplr"
Oct 30 13:11:14.874364 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:14.874339    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7hplr"
Oct 30 13:11:15.560249 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:15.560211    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-5rw2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 30 13:11:15.560446 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:15.560267    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h" podUID="35df4dfa-f04b-47b8-86cc-8d7758fcec5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 30 13:11:15.800049 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:15.800019    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7hplr"
Oct 30 13:11:16.560372 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:16.560327    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-5rw2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 30 13:11:16.560532 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:16.560402    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h" podUID="35df4dfa-f04b-47b8-86cc-8d7758fcec5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 30 13:11:16.678642 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:16.678591    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg"
Oct 30 13:11:16.691763 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:16.691735    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg"
Oct 30 13:11:16.805102 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:16.805070    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg"
Oct 30 13:11:18.561367 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:18.561325    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-5rw2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:11:18.561787 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:18.561397    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h" podUID="35df4dfa-f04b-47b8-86cc-8d7758fcec5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:19.603351 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:19.603307    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-5rw2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:11:19.603747 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:19.603377    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h" podUID="35df4dfa-f04b-47b8-86cc-8d7758fcec5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:19.863563 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:19.863477    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h"
Oct 30 13:11:19.864058 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:19.864027    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h"
Oct 30 13:11:19.864817 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:19.864800    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h"
Oct 30 13:11:21.166057 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:11:21.166013    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:11:22.039058 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:22.039022    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-568c86c4df-qqzql"]
Oct 30 13:11:22.045629 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:22.045605    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-568c86c4df-qqzql"
Oct 30 13:11:22.050267 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:22.050241    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqwks\" (UniqueName: \"kubernetes.io/projected/a6cde889-a80c-470e-86ea-1d930042a4ef-kube-api-access-bqwks\") pod \"authorino-568c86c4df-qqzql\" (UID: \"a6cde889-a80c-470e-86ea-1d930042a4ef\") " pod="kuadrant-system/authorino-568c86c4df-qqzql"
Oct 30 13:11:22.050397 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:22.050275    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a6cde889-a80c-470e-86ea-1d930042a4ef-tls-cert\") pod \"authorino-568c86c4df-qqzql\" (UID: \"a6cde889-a80c-470e-86ea-1d930042a4ef\") " pod="kuadrant-system/authorino-568c86c4df-qqzql"
Oct 30 13:11:22.065795 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:22.065772    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-sxnrz\""
Oct 30 13:11:22.066053 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:22.066034    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-568c86c4df-qqzql"]
Oct 30 13:11:22.066465 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:22.066447    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 30 13:11:22.151375 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:22.151335    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqwks\" (UniqueName: \"kubernetes.io/projected/a6cde889-a80c-470e-86ea-1d930042a4ef-kube-api-access-bqwks\") pod \"authorino-568c86c4df-qqzql\" (UID: \"a6cde889-a80c-470e-86ea-1d930042a4ef\") " pod="kuadrant-system/authorino-568c86c4df-qqzql"
Oct 30 13:11:22.151564 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:22.151382    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a6cde889-a80c-470e-86ea-1d930042a4ef-tls-cert\") pod \"authorino-568c86c4df-qqzql\" (UID: \"a6cde889-a80c-470e-86ea-1d930042a4ef\") " pod="kuadrant-system/authorino-568c86c4df-qqzql"
Oct 30 13:11:22.153954 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:22.153924    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a6cde889-a80c-470e-86ea-1d930042a4ef-tls-cert\") pod \"authorino-568c86c4df-qqzql\" (UID: \"a6cde889-a80c-470e-86ea-1d930042a4ef\") " pod="kuadrant-system/authorino-568c86c4df-qqzql"
Oct 30 13:11:22.178161 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:22.178122    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqwks\" (UniqueName: \"kubernetes.io/projected/a6cde889-a80c-470e-86ea-1d930042a4ef-kube-api-access-bqwks\") pod \"authorino-568c86c4df-qqzql\" (UID: \"a6cde889-a80c-470e-86ea-1d930042a4ef\") " pod="kuadrant-system/authorino-568c86c4df-qqzql"
Oct 30 13:11:22.359501 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:22.359407    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-568c86c4df-qqzql"
Oct 30 13:11:22.655709 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:22.655673    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-568c86c4df-qqzql"]
Oct 30 13:11:22.657733 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:11:22.657706    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6cde889_a80c_470e_86ea_1d930042a4ef.slice/crio-59b23ac5d52791759989b0817c4a276e878732356b90240c5a643a40fd75ba39 WatchSource:0}: Error finding container 59b23ac5d52791759989b0817c4a276e878732356b90240c5a643a40fd75ba39: Status 404 returned error can't find the container with id 59b23ac5d52791759989b0817c4a276e878732356b90240c5a643a40fd75ba39
Oct 30 13:11:22.833617 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:22.833572    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-568c86c4df-qqzql" event={"ID":"a6cde889-a80c-470e-86ea-1d930042a4ef","Type":"ContainerStarted","Data":"59b23ac5d52791759989b0817c4a276e878732356b90240c5a643a40fd75ba39"}
Oct 30 13:11:23.841318 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:23.841225    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-568c86c4df-qqzql" event={"ID":"a6cde889-a80c-470e-86ea-1d930042a4ef","Type":"ContainerStarted","Data":"2268a8db9c238c69fac031f92e8fa36133bf345eecd7a8335084d1aa934270e1"}
Oct 30 13:11:23.885964 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:23.885908    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-568c86c4df-qqzql" podStartSLOduration=0.979535341 podStartE2EDuration="1.885892014s" totalImagesPullingTime="906.356673ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:23.884541492 +0000 UTC m=+503.031808732" watchObservedRunningTime="2026-10-30 13:11:23.885892014 +0000 UTC m=+503.033159249"
Oct 30 13:11:24.323686 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:24.323644    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb"]
Oct 30 13:11:24.328311 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:24.328283    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb"
Oct 30 13:11:24.359970 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:24.359931    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb"]
Oct 30 13:11:24.371438 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:24.371408    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e97d49cb-0e82-4c03-8aeb-044f1c076234-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-9sgpb\" (UID: \"e97d49cb-0e82-4c03-8aeb-044f1c076234\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb"
Oct 30 13:11:24.371438 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:24.371440    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e97d49cb-0e82-4c03-8aeb-044f1c076234-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-9sgpb\" (UID: \"e97d49cb-0e82-4c03-8aeb-044f1c076234\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb"
Oct 30 13:11:24.371655 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:24.371472    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e97d49cb-0e82-4c03-8aeb-044f1c076234-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-9sgpb\" (UID: \"e97d49cb-0e82-4c03-8aeb-044f1c076234\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb"
Oct 30 13:11:24.371655 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:24.371589    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e97d49cb-0e82-4c03-8aeb-044f1c076234-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-9sgpb\" (UID: \"e97d49cb-0e82-4c03-8aeb-044f1c076234\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb"
Oct 30 13:11:24.371655 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:24.371631    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj4vw\" (UniqueName: \"kubernetes.io/projected/e97d49cb-0e82-4c03-8aeb-044f1c076234-kube-api-access-qj4vw\") pod \"maas-default-gateway-openshift-default-67bf986f99-9sgpb\" (UID: \"e97d49cb-0e82-4c03-8aeb-044f1c076234\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb"
Oct 30 13:11:24.371655 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:24.371649    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e97d49cb-0e82-4c03-8aeb-044f1c076234-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-9sgpb\" (UID: \"e97d49cb-0e82-4c03-8aeb-044f1c076234\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb"
Oct 30 13:11:24.371831 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:24.371670    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e97d49cb-0e82-4c03-8aeb-044f1c076234-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-9sgpb\" (UID: \"e97d49cb-0e82-4c03-8aeb-044f1c076234\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb"
Oct 30 13:11:24.371831 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:24.371792    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e97d49cb-0e82-4c03-8aeb-044f1c076234-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-9sgpb\" (UID: \"e97d49cb-0e82-4c03-8aeb-044f1c076234\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb"
Oct 30 13:11:24.371908 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:24.371834    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e97d49cb-0e82-4c03-8aeb-044f1c076234-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-9sgpb\" (UID: \"e97d49cb-0e82-4c03-8aeb-044f1c076234\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb"
Oct 30 13:11:24.472671 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:24.472620    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj4vw\" (UniqueName: \"kubernetes.io/projected/e97d49cb-0e82-4c03-8aeb-044f1c076234-kube-api-access-qj4vw\") pod \"maas-default-gateway-openshift-default-67bf986f99-9sgpb\" (UID: \"e97d49cb-0e82-4c03-8aeb-044f1c076234\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb"
Oct 30 13:11:24.472671 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:24.472677    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e97d49cb-0e82-4c03-8aeb-044f1c076234-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-9sgpb\" (UID: \"e97d49cb-0e82-4c03-8aeb-044f1c076234\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb"
Oct 30 13:11:24.472953 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:24.472714    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e97d49cb-0e82-4c03-8aeb-044f1c076234-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-9sgpb\" (UID: \"e97d49cb-0e82-4c03-8aeb-044f1c076234\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb"
Oct 30 13:11:24.472953 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:24.472761    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e97d49cb-0e82-4c03-8aeb-044f1c076234-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-9sgpb\" (UID: \"e97d49cb-0e82-4c03-8aeb-044f1c076234\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb"
Oct 30 13:11:24.472953 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:24.472838    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e97d49cb-0e82-4c03-8aeb-044f1c076234-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-9sgpb\" (UID: \"e97d49cb-0e82-4c03-8aeb-044f1c076234\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb"
Oct 30 13:11:24.472953 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:24.472891    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e97d49cb-0e82-4c03-8aeb-044f1c076234-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-9sgpb\" (UID: \"e97d49cb-0e82-4c03-8aeb-044f1c076234\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb"
Oct 30 13:11:24.472953 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:24.472919    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e97d49cb-0e82-4c03-8aeb-044f1c076234-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-9sgpb\" (UID: \"e97d49cb-0e82-4c03-8aeb-044f1c076234\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb"
Oct 30 13:11:24.473272 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:24.472975    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e97d49cb-0e82-4c03-8aeb-044f1c076234-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-9sgpb\" (UID: \"e97d49cb-0e82-4c03-8aeb-044f1c076234\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb"
Oct 30 13:11:24.473272 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:24.473023    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e97d49cb-0e82-4c03-8aeb-044f1c076234-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-9sgpb\" (UID: \"e97d49cb-0e82-4c03-8aeb-044f1c076234\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb"
Oct 30 13:11:24.473393 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:24.473285    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e97d49cb-0e82-4c03-8aeb-044f1c076234-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-9sgpb\" (UID: \"e97d49cb-0e82-4c03-8aeb-044f1c076234\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb"
Oct 30 13:11:24.473520 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:24.473492    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e97d49cb-0e82-4c03-8aeb-044f1c076234-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-9sgpb\" (UID: \"e97d49cb-0e82-4c03-8aeb-044f1c076234\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb"
Oct 30 13:11:24.473615 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:24.473566    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e97d49cb-0e82-4c03-8aeb-044f1c076234-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-9sgpb\" (UID: \"e97d49cb-0e82-4c03-8aeb-044f1c076234\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb"
Oct 30 13:11:24.473852 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:24.473814    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e97d49cb-0e82-4c03-8aeb-044f1c076234-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-9sgpb\" (UID: \"e97d49cb-0e82-4c03-8aeb-044f1c076234\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb"
Oct 30 13:11:24.473962 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:24.473942    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e97d49cb-0e82-4c03-8aeb-044f1c076234-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-9sgpb\" (UID: \"e97d49cb-0e82-4c03-8aeb-044f1c076234\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb"
Oct 30 13:11:24.476111 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:24.476085    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e97d49cb-0e82-4c03-8aeb-044f1c076234-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-9sgpb\" (UID: \"e97d49cb-0e82-4c03-8aeb-044f1c076234\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb"
Oct 30 13:11:24.476111 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:24.476099    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e97d49cb-0e82-4c03-8aeb-044f1c076234-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-9sgpb\" (UID: \"e97d49cb-0e82-4c03-8aeb-044f1c076234\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb"
Oct 30 13:11:24.510524 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:24.510492    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e97d49cb-0e82-4c03-8aeb-044f1c076234-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-9sgpb\" (UID: \"e97d49cb-0e82-4c03-8aeb-044f1c076234\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb"
Oct 30 13:11:24.511416 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:24.511374    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj4vw\" (UniqueName: \"kubernetes.io/projected/e97d49cb-0e82-4c03-8aeb-044f1c076234-kube-api-access-qj4vw\") pod \"maas-default-gateway-openshift-default-67bf986f99-9sgpb\" (UID: \"e97d49cb-0e82-4c03-8aeb-044f1c076234\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb"
Oct 30 13:11:24.661654 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:24.661509    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb"
Oct 30 13:11:25.001514 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:25.001236    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb"]
Oct 30 13:11:25.003917 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:11:25.003880    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode97d49cb_0e82_4c03_8aeb_044f1c076234.slice/crio-2905bde47151f8e64fe21de23057cc2a086db9e488f3a428f0602057b5eea29e WatchSource:0}: Error finding container 2905bde47151f8e64fe21de23057cc2a086db9e488f3a428f0602057b5eea29e: Status 404 returned error can't find the container with id 2905bde47151f8e64fe21de23057cc2a086db9e488f3a428f0602057b5eea29e
Oct 30 13:11:25.006749 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:25.006716    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:11:25.006836 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:25.006787    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:11:25.006836 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:25.006820    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:11:25.875780 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:25.874812    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb" event={"ID":"e97d49cb-0e82-4c03-8aeb-044f1c076234","Type":"ContainerStarted","Data":"13d4d7fc86791bf2c2b3602f64ed6365c622885be6ac05705b4ca5792d7a8f5c"}
Oct 30 13:11:25.875780 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:25.874880    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb" event={"ID":"e97d49cb-0e82-4c03-8aeb-044f1c076234","Type":"ContainerStarted","Data":"2905bde47151f8e64fe21de23057cc2a086db9e488f3a428f0602057b5eea29e"}
Oct 30 13:11:25.957154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:25.956687    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb" podStartSLOduration=1.956662578 podStartE2EDuration="1.956662578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:25.9436427 +0000 UTC m=+505.090909942" watchObservedRunningTime="2026-10-30 13:11:25.956662578 +0000 UTC m=+505.103929820"
Oct 30 13:11:26.661689 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:26.661650    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb"
Oct 30 13:11:26.663446 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:26.663416    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9sgpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 30 13:11:26.663599 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:26.663482    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb" podUID="e97d49cb-0e82-4c03-8aeb-044f1c076234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 30 13:11:27.662555 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:27.662513    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9sgpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 30 13:11:27.663070 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:27.662586    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb" podUID="e97d49cb-0e82-4c03-8aeb-044f1c076234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 30 13:11:28.662708 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:28.662670    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9sgpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 30 13:11:28.663203 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:28.662755    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb" podUID="e97d49cb-0e82-4c03-8aeb-044f1c076234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 30 13:11:29.662838 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:29.662784    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9sgpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 30 13:11:29.663392 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:29.662857    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb" podUID="e97d49cb-0e82-4c03-8aeb-044f1c076234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 30 13:11:30.662804 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:30.662756    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9sgpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 30 13:11:30.662994 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:30.662847    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb" podUID="e97d49cb-0e82-4c03-8aeb-044f1c076234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 30 13:11:31.662630 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:31.662591    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9sgpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 30 13:11:31.662818 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:31.662659    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb" podUID="e97d49cb-0e82-4c03-8aeb-044f1c076234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 30 13:11:32.662025 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:32.661992    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9sgpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 30 13:11:32.662451 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:32.662049    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb" podUID="e97d49cb-0e82-4c03-8aeb-044f1c076234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 30 13:11:33.666430 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:33.666395    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb"
Oct 30 13:11:33.666862 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:33.666696    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb"
Oct 30 13:11:33.667379 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:33.667364    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb"
Oct 30 13:11:51.166633 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:11:51.166589    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:11:54.756206 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:54.756058    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd"]
Oct 30 13:11:54.760433 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:54.760403    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd"
Oct 30 13:11:54.780059 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:54.780030    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-d6a10701-gw-openshift-default-dockercfg-bqfv6\""
Oct 30 13:11:54.796012 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:54.795976    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd"]
Oct 30 13:11:54.911876 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:54.911839    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6babd5ad-925c-4ac6-ab8a-9dffbb090259-istio-podinfo\") pod \"e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd\" (UID: \"6babd5ad-925c-4ac6-ab8a-9dffbb090259\") " pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd"
Oct 30 13:11:54.912061 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:54.911893    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6babd5ad-925c-4ac6-ab8a-9dffbb090259-istiod-ca-cert\") pod \"e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd\" (UID: \"6babd5ad-925c-4ac6-ab8a-9dffbb090259\") " pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd"
Oct 30 13:11:54.912061 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:54.911993    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6babd5ad-925c-4ac6-ab8a-9dffbb090259-istio-token\") pod \"e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd\" (UID: \"6babd5ad-925c-4ac6-ab8a-9dffbb090259\") " pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd"
Oct 30 13:11:54.912061 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:54.912023    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6babd5ad-925c-4ac6-ab8a-9dffbb090259-workload-socket\") pod \"e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd\" (UID: \"6babd5ad-925c-4ac6-ab8a-9dffbb090259\") " pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd"
Oct 30 13:11:54.912061 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:54.912048    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6babd5ad-925c-4ac6-ab8a-9dffbb090259-istio-data\") pod \"e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd\" (UID: \"6babd5ad-925c-4ac6-ab8a-9dffbb090259\") " pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd"
Oct 30 13:11:54.912265 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:54.912065    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6babd5ad-925c-4ac6-ab8a-9dffbb090259-workload-certs\") pod \"e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd\" (UID: \"6babd5ad-925c-4ac6-ab8a-9dffbb090259\") " pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd"
Oct 30 13:11:54.912265 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:54.912164    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvmw9\" (UniqueName: \"kubernetes.io/projected/6babd5ad-925c-4ac6-ab8a-9dffbb090259-kube-api-access-wvmw9\") pod \"e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd\" (UID: \"6babd5ad-925c-4ac6-ab8a-9dffbb090259\") " pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd"
Oct 30 13:11:54.912265 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:54.912221    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6babd5ad-925c-4ac6-ab8a-9dffbb090259-credential-socket\") pod \"e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd\" (UID: \"6babd5ad-925c-4ac6-ab8a-9dffbb090259\") " pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd"
Oct 30 13:11:54.912265 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:54.912246    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6babd5ad-925c-4ac6-ab8a-9dffbb090259-istio-envoy\") pod \"e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd\" (UID: \"6babd5ad-925c-4ac6-ab8a-9dffbb090259\") " pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd"
Oct 30 13:11:55.013197 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:55.013066    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6babd5ad-925c-4ac6-ab8a-9dffbb090259-istio-podinfo\") pod \"e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd\" (UID: \"6babd5ad-925c-4ac6-ab8a-9dffbb090259\") " pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd"
Oct 30 13:11:55.013197 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:55.013119    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6babd5ad-925c-4ac6-ab8a-9dffbb090259-istiod-ca-cert\") pod \"e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd\" (UID: \"6babd5ad-925c-4ac6-ab8a-9dffbb090259\") " pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd"
Oct 30 13:11:55.013197 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:55.013177    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6babd5ad-925c-4ac6-ab8a-9dffbb090259-istio-token\") pod \"e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd\" (UID: \"6babd5ad-925c-4ac6-ab8a-9dffbb090259\") " pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd"
Oct 30 13:11:55.013197 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:55.013196    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6babd5ad-925c-4ac6-ab8a-9dffbb090259-workload-socket\") pod \"e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd\" (UID: \"6babd5ad-925c-4ac6-ab8a-9dffbb090259\") " pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd"
Oct 30 13:11:55.013538 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:55.013216    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6babd5ad-925c-4ac6-ab8a-9dffbb090259-istio-data\") pod \"e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd\" (UID: \"6babd5ad-925c-4ac6-ab8a-9dffbb090259\") " pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd"
Oct 30 13:11:55.013538 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:55.013231    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6babd5ad-925c-4ac6-ab8a-9dffbb090259-workload-certs\") pod \"e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd\" (UID: \"6babd5ad-925c-4ac6-ab8a-9dffbb090259\") " pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd"
Oct 30 13:11:55.013538 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:55.013249    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvmw9\" (UniqueName: \"kubernetes.io/projected/6babd5ad-925c-4ac6-ab8a-9dffbb090259-kube-api-access-wvmw9\") pod \"e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd\" (UID: \"6babd5ad-925c-4ac6-ab8a-9dffbb090259\") " pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd"
Oct 30 13:11:55.013538 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:55.013285    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6babd5ad-925c-4ac6-ab8a-9dffbb090259-credential-socket\") pod \"e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd\" (UID: \"6babd5ad-925c-4ac6-ab8a-9dffbb090259\") " pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd"
Oct 30 13:11:55.013538 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:55.013503    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6babd5ad-925c-4ac6-ab8a-9dffbb090259-istio-envoy\") pod \"e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd\" (UID: \"6babd5ad-925c-4ac6-ab8a-9dffbb090259\") " pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd"
Oct 30 13:11:55.013781 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:55.013648    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6babd5ad-925c-4ac6-ab8a-9dffbb090259-workload-certs\") pod \"e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd\" (UID: \"6babd5ad-925c-4ac6-ab8a-9dffbb090259\") " pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd"
Oct 30 13:11:55.013781 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:55.013703    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6babd5ad-925c-4ac6-ab8a-9dffbb090259-workload-socket\") pod \"e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd\" (UID: \"6babd5ad-925c-4ac6-ab8a-9dffbb090259\") " pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd"
Oct 30 13:11:55.013781 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:55.013764    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6babd5ad-925c-4ac6-ab8a-9dffbb090259-istio-data\") pod \"e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd\" (UID: \"6babd5ad-925c-4ac6-ab8a-9dffbb090259\") " pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd"
Oct 30 13:11:55.013906 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:55.013864    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6babd5ad-925c-4ac6-ab8a-9dffbb090259-credential-socket\") pod \"e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd\" (UID: \"6babd5ad-925c-4ac6-ab8a-9dffbb090259\") " pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd"
Oct 30 13:11:55.013997 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:55.013980    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6babd5ad-925c-4ac6-ab8a-9dffbb090259-istiod-ca-cert\") pod \"e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd\" (UID: \"6babd5ad-925c-4ac6-ab8a-9dffbb090259\") " pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd"
Oct 30 13:11:55.015756 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:55.015726    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6babd5ad-925c-4ac6-ab8a-9dffbb090259-istio-envoy\") pod \"e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd\" (UID: \"6babd5ad-925c-4ac6-ab8a-9dffbb090259\") " pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd"
Oct 30 13:11:55.015951 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:55.015933    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6babd5ad-925c-4ac6-ab8a-9dffbb090259-istio-podinfo\") pod \"e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd\" (UID: \"6babd5ad-925c-4ac6-ab8a-9dffbb090259\") " pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd"
Oct 30 13:11:55.034429 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:55.034389    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6babd5ad-925c-4ac6-ab8a-9dffbb090259-istio-token\") pod \"e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd\" (UID: \"6babd5ad-925c-4ac6-ab8a-9dffbb090259\") " pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd"
Oct 30 13:11:55.034588 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:55.034530    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvmw9\" (UniqueName: \"kubernetes.io/projected/6babd5ad-925c-4ac6-ab8a-9dffbb090259-kube-api-access-wvmw9\") pod \"e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd\" (UID: \"6babd5ad-925c-4ac6-ab8a-9dffbb090259\") " pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd"
Oct 30 13:11:55.080292 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:55.080258    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd"
Oct 30 13:11:55.394693 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:55.394665    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd"]
Oct 30 13:11:55.397146 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:11:55.397100    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6babd5ad_925c_4ac6_ab8a_9dffbb090259.slice/crio-7967b8b2fd86b50618392e60502f8c718215928cd8634d42ef136cc80f1bd1f7 WatchSource:0}: Error finding container 7967b8b2fd86b50618392e60502f8c718215928cd8634d42ef136cc80f1bd1f7: Status 404 returned error can't find the container with id 7967b8b2fd86b50618392e60502f8c718215928cd8634d42ef136cc80f1bd1f7
Oct 30 13:11:55.399647 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:55.399609    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:11:55.399748 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:55.399682    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:11:55.399748 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:55.399711    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:11:56.048002 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:56.047971    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd" event={"ID":"6babd5ad-925c-4ac6-ab8a-9dffbb090259","Type":"ContainerStarted","Data":"01b42cc629afb4f4db97ef5a7f15af1bbc2ff77afb578ec3dae5cda8d0e2dc89"}
Oct 30 13:11:56.048525 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:56.048032    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd" event={"ID":"6babd5ad-925c-4ac6-ab8a-9dffbb090259","Type":"ContainerStarted","Data":"7967b8b2fd86b50618392e60502f8c718215928cd8634d42ef136cc80f1bd1f7"}
Oct 30 13:11:56.081082 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:56.081045    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd"
Oct 30 13:11:56.086467 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:56.086439    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd"
Oct 30 13:11:56.092981 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:56.092933    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd" podStartSLOduration=2.092919802 podStartE2EDuration="2.092919802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:56.091292612 +0000 UTC m=+535.238559864" watchObservedRunningTime="2026-10-30 13:11:56.092919802 +0000 UTC m=+535.240187043"
Oct 30 13:11:57.054636 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:57.054600    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd"
Oct 30 13:11:57.055488 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:57.055465    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd"
Oct 30 13:11:58.912405 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:58.912367    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d6a10701-84f4cccb8c-k4552"]
Oct 30 13:11:58.916417 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:58.916398    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d6a10701-84f4cccb8c-k4552"
Oct 30 13:11:58.935077 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:58.935046    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d6a10701-84f4cccb8c-k4552"]
Oct 30 13:11:58.945275 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:58.945234    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-d6a10701\""
Oct 30 13:11:58.945275 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:58.945238    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-d6a10701-dockercfg-q5nxk\""
Oct 30 13:11:59.056234 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:59.056198    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq856\" (UniqueName: \"kubernetes.io/projected/f4d7901b-2969-4fb4-9805-f87f7f2a82b1-kube-api-access-fq856\") pod \"payload-pre-processing-e2e-migrate-d6a10701-84f4cccb8c-k4552\" (UID: \"f4d7901b-2969-4fb4-9805-f87f7f2a82b1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d6a10701-84f4cccb8c-k4552"
Oct 30 13:11:59.056413 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:59.056265    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4d7901b-2969-4fb4-9805-f87f7f2a82b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d6a10701-84f4cccb8c-k4552\" (UID: \"f4d7901b-2969-4fb4-9805-f87f7f2a82b1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d6a10701-84f4cccb8c-k4552"
Oct 30 13:11:59.157483 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:59.157435    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq856\" (UniqueName: \"kubernetes.io/projected/f4d7901b-2969-4fb4-9805-f87f7f2a82b1-kube-api-access-fq856\") pod \"payload-pre-processing-e2e-migrate-d6a10701-84f4cccb8c-k4552\" (UID: \"f4d7901b-2969-4fb4-9805-f87f7f2a82b1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d6a10701-84f4cccb8c-k4552"
Oct 30 13:11:59.157651 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:59.157627    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4d7901b-2969-4fb4-9805-f87f7f2a82b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d6a10701-84f4cccb8c-k4552\" (UID: \"f4d7901b-2969-4fb4-9805-f87f7f2a82b1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d6a10701-84f4cccb8c-k4552"
Oct 30 13:11:59.158508 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:59.158482    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4d7901b-2969-4fb4-9805-f87f7f2a82b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d6a10701-84f4cccb8c-k4552\" (UID: \"f4d7901b-2969-4fb4-9805-f87f7f2a82b1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d6a10701-84f4cccb8c-k4552"
Oct 30 13:11:59.178631 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:59.178599    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq856\" (UniqueName: \"kubernetes.io/projected/f4d7901b-2969-4fb4-9805-f87f7f2a82b1-kube-api-access-fq856\") pod \"payload-pre-processing-e2e-migrate-d6a10701-84f4cccb8c-k4552\" (UID: \"f4d7901b-2969-4fb4-9805-f87f7f2a82b1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d6a10701-84f4cccb8c-k4552"
Oct 30 13:11:59.231466 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:59.231429    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d6a10701-84f4cccb8c-k4552"
Oct 30 13:11:59.528047 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:11:59.528016    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d6a10701-84f4cccb8c-k4552"]
Oct 30 13:11:59.529609 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:11:59.529571    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4d7901b_2969_4fb4_9805_f87f7f2a82b1.slice/crio-52d11bf9b3a99e4b66436accfd4c822f5f450e6174c23c66b5ec21e10ce8bed3 WatchSource:0}: Error finding container 52d11bf9b3a99e4b66436accfd4c822f5f450e6174c23c66b5ec21e10ce8bed3: Status 404 returned error can't find the container with id 52d11bf9b3a99e4b66436accfd4c822f5f450e6174c23c66b5ec21e10ce8bed3
Oct 30 13:12:00.075512 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:00.075474    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d6a10701-84f4cccb8c-k4552" event={"ID":"f4d7901b-2969-4fb4-9805-f87f7f2a82b1","Type":"ContainerStarted","Data":"52d11bf9b3a99e4b66436accfd4c822f5f450e6174c23c66b5ec21e10ce8bed3"}
Oct 30 13:12:00.338050 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:00.337964    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-662ed421-7dbbd45746-c5rj6"]
Oct 30 13:12:00.342027 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:00.341994    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-662ed421-7dbbd45746-c5rj6"
Oct 30 13:12:00.360993 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:00.360961    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-662ed421-7dbbd45746-c5rj6"]
Oct 30 13:12:00.370820 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:00.370764    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-662ed421\""
Oct 30 13:12:00.370820 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:00.370775    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-662ed421-dockercfg-xcqht\""
Oct 30 13:12:00.387579 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:00.387550    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-662ed421-5b6df6999f-b69dp"]
Oct 30 13:12:00.396458 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:00.396425    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-662ed421-5b6df6999f-b69dp"
Oct 30 13:12:00.408585 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:00.408505    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-662ed421-5b6df6999f-b69dp"]
Oct 30 13:12:00.473003 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:00.472952    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdwcz\" (UniqueName: \"kubernetes.io/projected/2cc2a92b-d546-4cca-a0a0-fd23d881a30b-kube-api-access-kdwcz\") pod \"payload-processing-e2e-ait-662ed421-7dbbd45746-c5rj6\" (UID: \"2cc2a92b-d546-4cca-a0a0-fd23d881a30b\") " pod="openshift-ingress/payload-processing-e2e-ait-662ed421-7dbbd45746-c5rj6"
Oct 30 13:12:00.473515 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:00.473073    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cc2a92b-d546-4cca-a0a0-fd23d881a30b-plugins-config-volume\") pod \"payload-processing-e2e-ait-662ed421-7dbbd45746-c5rj6\" (UID: \"2cc2a92b-d546-4cca-a0a0-fd23d881a30b\") " pod="openshift-ingress/payload-processing-e2e-ait-662ed421-7dbbd45746-c5rj6"
Oct 30 13:12:00.573950 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:00.573903    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khgpx\" (UniqueName: \"kubernetes.io/projected/22ab0145-a9a0-41af-b06a-466519bccd0e-kube-api-access-khgpx\") pod \"payload-pre-processing-e2e-ait-662ed421-5b6df6999f-b69dp\" (UID: \"22ab0145-a9a0-41af-b06a-466519bccd0e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-662ed421-5b6df6999f-b69dp"
Oct 30 13:12:00.574145 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:00.574023    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22ab0145-a9a0-41af-b06a-466519bccd0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-662ed421-5b6df6999f-b69dp\" (UID: \"22ab0145-a9a0-41af-b06a-466519bccd0e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-662ed421-5b6df6999f-b69dp"
Oct 30 13:12:00.574145 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:00.574106    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cc2a92b-d546-4cca-a0a0-fd23d881a30b-plugins-config-volume\") pod \"payload-processing-e2e-ait-662ed421-7dbbd45746-c5rj6\" (UID: \"2cc2a92b-d546-4cca-a0a0-fd23d881a30b\") " pod="openshift-ingress/payload-processing-e2e-ait-662ed421-7dbbd45746-c5rj6"
Oct 30 13:12:00.574273 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:00.574256    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdwcz\" (UniqueName: \"kubernetes.io/projected/2cc2a92b-d546-4cca-a0a0-fd23d881a30b-kube-api-access-kdwcz\") pod \"payload-processing-e2e-ait-662ed421-7dbbd45746-c5rj6\" (UID: \"2cc2a92b-d546-4cca-a0a0-fd23d881a30b\") " pod="openshift-ingress/payload-processing-e2e-ait-662ed421-7dbbd45746-c5rj6"
Oct 30 13:12:00.574930 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:00.574900    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cc2a92b-d546-4cca-a0a0-fd23d881a30b-plugins-config-volume\") pod \"payload-processing-e2e-ait-662ed421-7dbbd45746-c5rj6\" (UID: \"2cc2a92b-d546-4cca-a0a0-fd23d881a30b\") " pod="openshift-ingress/payload-processing-e2e-ait-662ed421-7dbbd45746-c5rj6"
Oct 30 13:12:00.600520 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:00.600420    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdwcz\" (UniqueName: \"kubernetes.io/projected/2cc2a92b-d546-4cca-a0a0-fd23d881a30b-kube-api-access-kdwcz\") pod \"payload-processing-e2e-ait-662ed421-7dbbd45746-c5rj6\" (UID: \"2cc2a92b-d546-4cca-a0a0-fd23d881a30b\") " pod="openshift-ingress/payload-processing-e2e-ait-662ed421-7dbbd45746-c5rj6"
Oct 30 13:12:00.661350 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:00.661308    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-662ed421-7dbbd45746-c5rj6"
Oct 30 13:12:00.675290 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:00.675160    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22ab0145-a9a0-41af-b06a-466519bccd0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-662ed421-5b6df6999f-b69dp\" (UID: \"22ab0145-a9a0-41af-b06a-466519bccd0e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-662ed421-5b6df6999f-b69dp"
Oct 30 13:12:00.675378 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:00.675351    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khgpx\" (UniqueName: \"kubernetes.io/projected/22ab0145-a9a0-41af-b06a-466519bccd0e-kube-api-access-khgpx\") pod \"payload-pre-processing-e2e-ait-662ed421-5b6df6999f-b69dp\" (UID: \"22ab0145-a9a0-41af-b06a-466519bccd0e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-662ed421-5b6df6999f-b69dp"
Oct 30 13:12:00.676431 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:00.676401    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22ab0145-a9a0-41af-b06a-466519bccd0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-662ed421-5b6df6999f-b69dp\" (UID: \"22ab0145-a9a0-41af-b06a-466519bccd0e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-662ed421-5b6df6999f-b69dp"
Oct 30 13:12:00.717659 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:00.717597    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khgpx\" (UniqueName: \"kubernetes.io/projected/22ab0145-a9a0-41af-b06a-466519bccd0e-kube-api-access-khgpx\") pod \"payload-pre-processing-e2e-ait-662ed421-5b6df6999f-b69dp\" (UID: \"22ab0145-a9a0-41af-b06a-466519bccd0e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-662ed421-5b6df6999f-b69dp"
Oct 30 13:12:00.758195 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:00.756327    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-662ed421-84ddd86fb7-lz7dl"]
Oct 30 13:12:00.765495 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:00.765473    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-662ed421-84ddd86fb7-lz7dl"
Oct 30 13:12:00.799581 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:00.799548    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-662ed421-84ddd86fb7-lz7dl"]
Oct 30 13:12:00.808727 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:00.808700    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gzx7f\""
Oct 30 13:12:00.810006 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:00.809977    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:12:00.810292 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:00.810274    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:12:00.882267 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:00.882167    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a54fa560-1acc-4d98-adea-3648e998da77-openshift-service-ca\") pod \"maas-api-e2e-ait-662ed421-84ddd86fb7-lz7dl\" (UID: \"a54fa560-1acc-4d98-adea-3648e998da77\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-662ed421-84ddd86fb7-lz7dl"
Oct 30 13:12:00.882447 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:00.882269    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a54fa560-1acc-4d98-adea-3648e998da77-metrics-certs\") pod \"maas-api-e2e-ait-662ed421-84ddd86fb7-lz7dl\" (UID: \"a54fa560-1acc-4d98-adea-3648e998da77\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-662ed421-84ddd86fb7-lz7dl"
Oct 30 13:12:00.882447 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:00.882339    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtb26\" (UniqueName: \"kubernetes.io/projected/a54fa560-1acc-4d98-adea-3648e998da77-kube-api-access-rtb26\") pod \"maas-api-e2e-ait-662ed421-84ddd86fb7-lz7dl\" (UID: \"a54fa560-1acc-4d98-adea-3648e998da77\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-662ed421-84ddd86fb7-lz7dl"
Oct 30 13:12:00.882447 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:00.882428    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a54fa560-1acc-4d98-adea-3648e998da77-maas-api-tls\") pod \"maas-api-e2e-ait-662ed421-84ddd86fb7-lz7dl\" (UID: \"a54fa560-1acc-4d98-adea-3648e998da77\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-662ed421-84ddd86fb7-lz7dl"
Oct 30 13:12:00.887265 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:00.887235    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 13:12:00.887438 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:00.887280    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-662ed421-serving-cert\""
Oct 30 13:12:00.983626 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:00.983591    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-662ed421-7dbbd45746-c5rj6"]
Oct 30 13:12:00.984841 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:00.984815    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a54fa560-1acc-4d98-adea-3648e998da77-maas-api-tls\") pod \"maas-api-e2e-ait-662ed421-84ddd86fb7-lz7dl\" (UID: \"a54fa560-1acc-4d98-adea-3648e998da77\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-662ed421-84ddd86fb7-lz7dl"
Oct 30 13:12:00.984961 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:00.984875    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a54fa560-1acc-4d98-adea-3648e998da77-openshift-service-ca\") pod \"maas-api-e2e-ait-662ed421-84ddd86fb7-lz7dl\" (UID: \"a54fa560-1acc-4d98-adea-3648e998da77\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-662ed421-84ddd86fb7-lz7dl"
Oct 30 13:12:00.984961 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:00.984932    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a54fa560-1acc-4d98-adea-3648e998da77-metrics-certs\") pod \"maas-api-e2e-ait-662ed421-84ddd86fb7-lz7dl\" (UID: \"a54fa560-1acc-4d98-adea-3648e998da77\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-662ed421-84ddd86fb7-lz7dl"
Oct 30 13:12:00.985245 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:00.984985    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtb26\" (UniqueName: \"kubernetes.io/projected/a54fa560-1acc-4d98-adea-3648e998da77-kube-api-access-rtb26\") pod \"maas-api-e2e-ait-662ed421-84ddd86fb7-lz7dl\" (UID: \"a54fa560-1acc-4d98-adea-3648e998da77\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-662ed421-84ddd86fb7-lz7dl"
Oct 30 13:12:00.988725 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:12:00.988694    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cc2a92b_d546_4cca_a0a0_fd23d881a30b.slice/crio-2fba4d8c4377a4b7c27fc8912a79b35fb9be4a19e6ea0fe243fd0473094c634c WatchSource:0}: Error finding container 2fba4d8c4377a4b7c27fc8912a79b35fb9be4a19e6ea0fe243fd0473094c634c: Status 404 returned error can't find the container with id 2fba4d8c4377a4b7c27fc8912a79b35fb9be4a19e6ea0fe243fd0473094c634c
Oct 30 13:12:00.999853 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:00.999829    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:12:00.999980 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:00.999921    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 13:12:01.000049 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:00.999989    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-662ed421-serving-cert\""
Oct 30 13:12:01.006069 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:01.006044    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a54fa560-1acc-4d98-adea-3648e998da77-openshift-service-ca\") pod \"maas-api-e2e-ait-662ed421-84ddd86fb7-lz7dl\" (UID: \"a54fa560-1acc-4d98-adea-3648e998da77\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-662ed421-84ddd86fb7-lz7dl"
Oct 30 13:12:01.008522 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:01.008493    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a54fa560-1acc-4d98-adea-3648e998da77-maas-api-tls\") pod \"maas-api-e2e-ait-662ed421-84ddd86fb7-lz7dl\" (UID: \"a54fa560-1acc-4d98-adea-3648e998da77\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-662ed421-84ddd86fb7-lz7dl"
Oct 30 13:12:01.008639 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:01.008525    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a54fa560-1acc-4d98-adea-3648e998da77-metrics-certs\") pod \"maas-api-e2e-ait-662ed421-84ddd86fb7-lz7dl\" (UID: \"a54fa560-1acc-4d98-adea-3648e998da77\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-662ed421-84ddd86fb7-lz7dl"
Oct 30 13:12:01.015218 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:01.015194    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-662ed421-5b6df6999f-b69dp"
Oct 30 13:12:01.017952 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:01.017935    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:12:01.026814 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:01.026789    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtb26\" (UniqueName: \"kubernetes.io/projected/a54fa560-1acc-4d98-adea-3648e998da77-kube-api-access-rtb26\") pod \"maas-api-e2e-ait-662ed421-84ddd86fb7-lz7dl\" (UID: \"a54fa560-1acc-4d98-adea-3648e998da77\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-662ed421-84ddd86fb7-lz7dl"
Oct 30 13:12:01.086271 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:01.086237    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-662ed421-7dbbd45746-c5rj6" event={"ID":"2cc2a92b-d546-4cca-a0a0-fd23d881a30b","Type":"ContainerStarted","Data":"2fba4d8c4377a4b7c27fc8912a79b35fb9be4a19e6ea0fe243fd0473094c634c"}
Oct 30 13:12:01.103530 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:01.103482    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gzx7f\""
Oct 30 13:12:01.109074 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:01.108761    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-662ed421-84ddd86fb7-lz7dl"
Oct 30 13:12:01.355342 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:12:01.355302    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22ab0145_a9a0_41af_b06a_466519bccd0e.slice/crio-6feced7b29154bc7468b1f6e3d30710ad072e3e9e4cb424c49cfe6e8d4c2419a WatchSource:0}: Error finding container 6feced7b29154bc7468b1f6e3d30710ad072e3e9e4cb424c49cfe6e8d4c2419a: Status 404 returned error can't find the container with id 6feced7b29154bc7468b1f6e3d30710ad072e3e9e4cb424c49cfe6e8d4c2419a
Oct 30 13:12:01.359822 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:01.359793    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-662ed421-5b6df6999f-b69dp"]
Oct 30 13:12:01.363741 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:01.363708    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fc4197ef-5f7fb55b75-mps2d"]
Oct 30 13:12:01.369522 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:01.369479    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc4197ef-5f7fb55b75-mps2d"
Oct 30 13:12:01.374729 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:01.373752    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fc4197ef-5f7fb55b75-mps2d"]
Oct 30 13:12:01.386694 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:01.386667    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-fc4197ef-dockercfg-sx4hp\""
Oct 30 13:12:01.417922 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:01.417876    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-fc4197ef\""
Oct 30 13:12:01.426571 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:01.426483    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-662ed421-84ddd86fb7-lz7dl"]
Oct 30 13:12:01.463417 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:12:01.463280    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda54fa560_1acc_4d98_adea_3648e998da77.slice/crio-427ba00fa0d77e3958e709e6752e507e3cf449e1ad274d5c23689b5c066a81c0 WatchSource:0}: Error finding container 427ba00fa0d77e3958e709e6752e507e3cf449e1ad274d5c23689b5c066a81c0: Status 404 returned error can't find the container with id 427ba00fa0d77e3958e709e6752e507e3cf449e1ad274d5c23689b5c066a81c0
Oct 30 13:12:01.490076 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:01.490005    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e9eef72-38bd-4fea-bd12-36d611a3c933-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-fc4197ef-5f7fb55b75-mps2d\" (UID: \"3e9eef72-38bd-4fea-bd12-36d611a3c933\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fc4197ef-5f7fb55b75-mps2d"
Oct 30 13:12:01.490076 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:01.490077    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzr8b\" (UniqueName: \"kubernetes.io/projected/3e9eef72-38bd-4fea-bd12-36d611a3c933-kube-api-access-tzr8b\") pod \"payload-pre-processing-e2e-ait-fc4197ef-5f7fb55b75-mps2d\" (UID: \"3e9eef72-38bd-4fea-bd12-36d611a3c933\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fc4197ef-5f7fb55b75-mps2d"
Oct 30 13:12:01.591543 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:01.591506    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e9eef72-38bd-4fea-bd12-36d611a3c933-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-fc4197ef-5f7fb55b75-mps2d\" (UID: \"3e9eef72-38bd-4fea-bd12-36d611a3c933\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fc4197ef-5f7fb55b75-mps2d"
Oct 30 13:12:01.591743 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:01.591566    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzr8b\" (UniqueName: \"kubernetes.io/projected/3e9eef72-38bd-4fea-bd12-36d611a3c933-kube-api-access-tzr8b\") pod \"payload-pre-processing-e2e-ait-fc4197ef-5f7fb55b75-mps2d\" (UID: \"3e9eef72-38bd-4fea-bd12-36d611a3c933\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fc4197ef-5f7fb55b75-mps2d"
Oct 30 13:12:01.592623 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:01.592582    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e9eef72-38bd-4fea-bd12-36d611a3c933-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-fc4197ef-5f7fb55b75-mps2d\" (UID: \"3e9eef72-38bd-4fea-bd12-36d611a3c933\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fc4197ef-5f7fb55b75-mps2d"
Oct 30 13:12:01.612483 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:01.612415    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzr8b\" (UniqueName: \"kubernetes.io/projected/3e9eef72-38bd-4fea-bd12-36d611a3c933-kube-api-access-tzr8b\") pod \"payload-pre-processing-e2e-ait-fc4197ef-5f7fb55b75-mps2d\" (UID: \"3e9eef72-38bd-4fea-bd12-36d611a3c933\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fc4197ef-5f7fb55b75-mps2d"
Oct 30 13:12:01.690434 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:01.690341    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc4197ef-5f7fb55b75-mps2d"
Oct 30 13:12:02.030150 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:02.029980    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fc4197ef-5f7fb55b75-mps2d"]
Oct 30 13:12:02.035872 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:12:02.035834    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e9eef72_38bd_4fea_bd12_36d611a3c933.slice/crio-a1c0da2f0547b54d3536dca8a972e4de601bc66c61ef7b0ad5cdf887f522d5a2 WatchSource:0}: Error finding container a1c0da2f0547b54d3536dca8a972e4de601bc66c61ef7b0ad5cdf887f522d5a2: Status 404 returned error can't find the container with id a1c0da2f0547b54d3536dca8a972e4de601bc66c61ef7b0ad5cdf887f522d5a2
Oct 30 13:12:02.121564 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:02.121469    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc4197ef-5f7fb55b75-mps2d" event={"ID":"3e9eef72-38bd-4fea-bd12-36d611a3c933","Type":"ContainerStarted","Data":"a1c0da2f0547b54d3536dca8a972e4de601bc66c61ef7b0ad5cdf887f522d5a2"}
Oct 30 13:12:02.124113 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:02.124030    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-662ed421-84ddd86fb7-lz7dl" event={"ID":"a54fa560-1acc-4d98-adea-3648e998da77","Type":"ContainerStarted","Data":"427ba00fa0d77e3958e709e6752e507e3cf449e1ad274d5c23689b5c066a81c0"}
Oct 30 13:12:02.126467 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:02.126354    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-662ed421-5b6df6999f-b69dp" event={"ID":"22ab0145-a9a0-41af-b06a-466519bccd0e","Type":"ContainerStarted","Data":"6feced7b29154bc7468b1f6e3d30710ad072e3e9e4cb424c49cfe6e8d4c2419a"}
Oct 30 13:12:03.183811 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:03.183778    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d6a10701-84f4cccb8c-k4552" event={"ID":"f4d7901b-2969-4fb4-9805-f87f7f2a82b1","Type":"ContainerStarted","Data":"0b235bcaf12a9776f4d223281af5ae366f76270c37d969bc5a5a6f908a9803a0"}
Oct 30 13:12:03.184470 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:03.184452    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d6a10701-84f4cccb8c-k4552"
Oct 30 13:12:03.261291 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:03.257927    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d6a10701-84f4cccb8c-k4552" podStartSLOduration=2.32496996 podStartE2EDuration="5.257904662s" totalImagesPullingTime="2.932934702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:03.252912901 +0000 UTC m=+542.400180142" watchObservedRunningTime="2026-10-30 13:12:03.257904662 +0000 UTC m=+542.405171904"
Oct 30 13:12:03.389941 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:03.388447    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-deb5d83f-9bcf86c6c-87gkt"]
Oct 30 13:12:03.393064 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:03.393040    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-deb5d83f-9bcf86c6c-87gkt"
Oct 30 13:12:03.413162 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:03.412558    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-deb5d83f\""
Oct 30 13:12:03.413162 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:03.412964    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-deb5d83f-9bcf86c6c-87gkt"]
Oct 30 13:12:03.467697 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:03.467566    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx9bk\" (UniqueName: \"kubernetes.io/projected/852636f3-4027-4ba9-a54a-9bb97658752a-kube-api-access-dx9bk\") pod \"payload-pre-processing-e2e-ait-deb5d83f-9bcf86c6c-87gkt\" (UID: \"852636f3-4027-4ba9-a54a-9bb97658752a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-deb5d83f-9bcf86c6c-87gkt"
Oct 30 13:12:03.467697 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:03.467648    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/852636f3-4027-4ba9-a54a-9bb97658752a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-deb5d83f-9bcf86c6c-87gkt\" (UID: \"852636f3-4027-4ba9-a54a-9bb97658752a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-deb5d83f-9bcf86c6c-87gkt"
Oct 30 13:12:03.582169 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:03.580811    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx9bk\" (UniqueName: \"kubernetes.io/projected/852636f3-4027-4ba9-a54a-9bb97658752a-kube-api-access-dx9bk\") pod \"payload-pre-processing-e2e-ait-deb5d83f-9bcf86c6c-87gkt\" (UID: \"852636f3-4027-4ba9-a54a-9bb97658752a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-deb5d83f-9bcf86c6c-87gkt"
Oct 30 13:12:03.582169 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:03.580899    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/852636f3-4027-4ba9-a54a-9bb97658752a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-deb5d83f-9bcf86c6c-87gkt\" (UID: \"852636f3-4027-4ba9-a54a-9bb97658752a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-deb5d83f-9bcf86c6c-87gkt"
Oct 30 13:12:03.582169 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:03.581855    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/852636f3-4027-4ba9-a54a-9bb97658752a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-deb5d83f-9bcf86c6c-87gkt\" (UID: \"852636f3-4027-4ba9-a54a-9bb97658752a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-deb5d83f-9bcf86c6c-87gkt"
Oct 30 13:12:03.613269 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:03.612871    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx9bk\" (UniqueName: \"kubernetes.io/projected/852636f3-4027-4ba9-a54a-9bb97658752a-kube-api-access-dx9bk\") pod \"payload-pre-processing-e2e-ait-deb5d83f-9bcf86c6c-87gkt\" (UID: \"852636f3-4027-4ba9-a54a-9bb97658752a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-deb5d83f-9bcf86c6c-87gkt"
Oct 30 13:12:03.724351 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:03.723302    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-deb5d83f-9bcf86c6c-87gkt"
Oct 30 13:12:03.737824 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:03.737772    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c"]
Oct 30 13:12:03.771170 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:03.771115    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c"]
Oct 30 13:12:03.771391 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:03.771273    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c"
Oct 30 13:12:03.787669 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:03.787646    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-fb3b54cb-openshift-default-dockercfg-qn4nd\""
Oct 30 13:12:03.889551 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:03.888896    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7709620c-73e4-4a9e-8b06-ef35865ea56d-workload-certs\") pod \"e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c\" (UID: \"7709620c-73e4-4a9e-8b06-ef35865ea56d\") " pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c"
Oct 30 13:12:03.889551 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:03.888952    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7709620c-73e4-4a9e-8b06-ef35865ea56d-istio-token\") pod \"e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c\" (UID: \"7709620c-73e4-4a9e-8b06-ef35865ea56d\") " pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c"
Oct 30 13:12:03.889551 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:03.888987    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7709620c-73e4-4a9e-8b06-ef35865ea56d-istio-data\") pod \"e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c\" (UID: \"7709620c-73e4-4a9e-8b06-ef35865ea56d\") " pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c"
Oct 30 13:12:03.889551 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:03.889034    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb82v\" (UniqueName: \"kubernetes.io/projected/7709620c-73e4-4a9e-8b06-ef35865ea56d-kube-api-access-rb82v\") pod \"e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c\" (UID: \"7709620c-73e4-4a9e-8b06-ef35865ea56d\") " pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c"
Oct 30 13:12:03.889551 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:03.889084    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7709620c-73e4-4a9e-8b06-ef35865ea56d-istiod-ca-cert\") pod \"e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c\" (UID: \"7709620c-73e4-4a9e-8b06-ef35865ea56d\") " pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c"
Oct 30 13:12:03.889551 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:03.889159    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7709620c-73e4-4a9e-8b06-ef35865ea56d-workload-socket\") pod \"e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c\" (UID: \"7709620c-73e4-4a9e-8b06-ef35865ea56d\") " pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c"
Oct 30 13:12:03.889551 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:03.889238    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7709620c-73e4-4a9e-8b06-ef35865ea56d-credential-socket\") pod \"e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c\" (UID: \"7709620c-73e4-4a9e-8b06-ef35865ea56d\") " pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c"
Oct 30 13:12:03.889551 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:03.889270    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7709620c-73e4-4a9e-8b06-ef35865ea56d-istio-envoy\") pod \"e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c\" (UID: \"7709620c-73e4-4a9e-8b06-ef35865ea56d\") " pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c"
Oct 30 13:12:03.889551 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:03.889312    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7709620c-73e4-4a9e-8b06-ef35865ea56d-istio-podinfo\") pod \"e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c\" (UID: \"7709620c-73e4-4a9e-8b06-ef35865ea56d\") " pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c"
Oct 30 13:12:03.992253 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:03.991641    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7709620c-73e4-4a9e-8b06-ef35865ea56d-credential-socket\") pod \"e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c\" (UID: \"7709620c-73e4-4a9e-8b06-ef35865ea56d\") " pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c"
Oct 30 13:12:03.992253 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:03.991689    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7709620c-73e4-4a9e-8b06-ef35865ea56d-istio-envoy\") pod \"e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c\" (UID: \"7709620c-73e4-4a9e-8b06-ef35865ea56d\") " pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c"
Oct 30 13:12:03.992253 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:03.991715    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7709620c-73e4-4a9e-8b06-ef35865ea56d-istio-podinfo\") pod \"e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c\" (UID: \"7709620c-73e4-4a9e-8b06-ef35865ea56d\") " pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c"
Oct 30 13:12:03.992253 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:03.991778    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7709620c-73e4-4a9e-8b06-ef35865ea56d-workload-certs\") pod \"e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c\" (UID: \"7709620c-73e4-4a9e-8b06-ef35865ea56d\") " pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c"
Oct 30 13:12:03.992253 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:03.991793    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7709620c-73e4-4a9e-8b06-ef35865ea56d-istio-token\") pod \"e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c\" (UID: \"7709620c-73e4-4a9e-8b06-ef35865ea56d\") " pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c"
Oct 30 13:12:03.992253 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:03.991810    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7709620c-73e4-4a9e-8b06-ef35865ea56d-istio-data\") pod \"e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c\" (UID: \"7709620c-73e4-4a9e-8b06-ef35865ea56d\") " pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c"
Oct 30 13:12:03.992253 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:03.991851    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb82v\" (UniqueName: \"kubernetes.io/projected/7709620c-73e4-4a9e-8b06-ef35865ea56d-kube-api-access-rb82v\") pod \"e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c\" (UID: \"7709620c-73e4-4a9e-8b06-ef35865ea56d\") " pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c"
Oct 30 13:12:03.992253 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:03.991899    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7709620c-73e4-4a9e-8b06-ef35865ea56d-istiod-ca-cert\") pod \"e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c\" (UID: \"7709620c-73e4-4a9e-8b06-ef35865ea56d\") " pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c"
Oct 30 13:12:03.992253 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:03.991962    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7709620c-73e4-4a9e-8b06-ef35865ea56d-workload-socket\") pod \"e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c\" (UID: \"7709620c-73e4-4a9e-8b06-ef35865ea56d\") " pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c"
Oct 30 13:12:03.993094 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:03.992624    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7709620c-73e4-4a9e-8b06-ef35865ea56d-workload-certs\") pod \"e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c\" (UID: \"7709620c-73e4-4a9e-8b06-ef35865ea56d\") " pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c"
Oct 30 13:12:03.993094 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:03.992903    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7709620c-73e4-4a9e-8b06-ef35865ea56d-credential-socket\") pod \"e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c\" (UID: \"7709620c-73e4-4a9e-8b06-ef35865ea56d\") " pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c"
Oct 30 13:12:03.994538 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:03.993578    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7709620c-73e4-4a9e-8b06-ef35865ea56d-workload-socket\") pod \"e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c\" (UID: \"7709620c-73e4-4a9e-8b06-ef35865ea56d\") " pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c"
Oct 30 13:12:03.994538 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:03.993821    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7709620c-73e4-4a9e-8b06-ef35865ea56d-istio-data\") pod \"e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c\" (UID: \"7709620c-73e4-4a9e-8b06-ef35865ea56d\") " pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c"
Oct 30 13:12:03.994538 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:03.994420    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7709620c-73e4-4a9e-8b06-ef35865ea56d-istiod-ca-cert\") pod \"e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c\" (UID: \"7709620c-73e4-4a9e-8b06-ef35865ea56d\") " pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c"
Oct 30 13:12:04.068188 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:04.068113    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7709620c-73e4-4a9e-8b06-ef35865ea56d-istio-podinfo\") pod \"e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c\" (UID: \"7709620c-73e4-4a9e-8b06-ef35865ea56d\") " pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c"
Oct 30 13:12:04.068362 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:04.068276    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb82v\" (UniqueName: \"kubernetes.io/projected/7709620c-73e4-4a9e-8b06-ef35865ea56d-kube-api-access-rb82v\") pod \"e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c\" (UID: \"7709620c-73e4-4a9e-8b06-ef35865ea56d\") " pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c"
Oct 30 13:12:04.068667 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:04.068644    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7709620c-73e4-4a9e-8b06-ef35865ea56d-istio-envoy\") pod \"e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c\" (UID: \"7709620c-73e4-4a9e-8b06-ef35865ea56d\") " pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c"
Oct 30 13:12:04.070079 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:04.070041    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7709620c-73e4-4a9e-8b06-ef35865ea56d-istio-token\") pod \"e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c\" (UID: \"7709620c-73e4-4a9e-8b06-ef35865ea56d\") " pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c"
Oct 30 13:12:04.157680 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:04.157343    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c"
Oct 30 13:12:04.158744 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:04.158705    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-deb5d83f-9bcf86c6c-87gkt"]
Oct 30 13:12:04.167631 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:12:04.167597    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod852636f3_4027_4ba9_a54a_9bb97658752a.slice/crio-647160cdfa14da20a69376655a0899f7687ed00c611f6ae854b79b544a294e90 WatchSource:0}: Error finding container 647160cdfa14da20a69376655a0899f7687ed00c611f6ae854b79b544a294e90: Status 404 returned error can't find the container with id 647160cdfa14da20a69376655a0899f7687ed00c611f6ae854b79b544a294e90
Oct 30 13:12:04.196065 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:04.195983    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-deb5d83f-9bcf86c6c-87gkt" event={"ID":"852636f3-4027-4ba9-a54a-9bb97658752a","Type":"ContainerStarted","Data":"647160cdfa14da20a69376655a0899f7687ed00c611f6ae854b79b544a294e90"}
Oct 30 13:12:04.201282 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:04.201149    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc4197ef-5f7fb55b75-mps2d" event={"ID":"3e9eef72-38bd-4fea-bd12-36d611a3c933","Type":"ContainerStarted","Data":"5486bf167d158788f59105dd5d6d4e1e73a7842e7e87593e96e7e17692268b4b"}
Oct 30 13:12:04.201282 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:04.201235    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc4197ef-5f7fb55b75-mps2d"
Oct 30 13:12:04.247706 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:04.246540    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc4197ef-5f7fb55b75-mps2d" podStartSLOduration=2.107662524 podStartE2EDuration="3.246517053s" totalImagesPullingTime="1.138854529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:04.243831687 +0000 UTC m=+543.391098951" watchObservedRunningTime="2026-10-30 13:12:04.246517053 +0000 UTC m=+543.393784301"
Oct 30 13:12:04.508847 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:04.508814    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c"]
Oct 30 13:12:04.510156 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:12:04.510108    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7709620c_73e4_4a9e_8b06_ef35865ea56d.slice/crio-486b2b6bec8b5202c3147e355f738f6a59988bf5545d3e04a700301bcaeb104e WatchSource:0}: Error finding container 486b2b6bec8b5202c3147e355f738f6a59988bf5545d3e04a700301bcaeb104e: Status 404 returned error can't find the container with id 486b2b6bec8b5202c3147e355f738f6a59988bf5545d3e04a700301bcaeb104e
Oct 30 13:12:04.513206 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:04.513097    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:12:04.513427 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:04.513398    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:12:04.513517 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:04.513457    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:12:05.210244 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:05.209690    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-662ed421-5b6df6999f-b69dp" event={"ID":"22ab0145-a9a0-41af-b06a-466519bccd0e","Type":"ContainerStarted","Data":"efcb318f841fc6db5cc49755822d3d407140c41599dcc3701fde49f5d71d9a84"}
Oct 30 13:12:05.210244 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:05.209902    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-662ed421-5b6df6999f-b69dp"
Oct 30 13:12:05.212012 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:05.211985    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-deb5d83f-9bcf86c6c-87gkt" event={"ID":"852636f3-4027-4ba9-a54a-9bb97658752a","Type":"ContainerStarted","Data":"13a93e1af4cf082502d0be81c3665ebf71b9af6b5a93f236eaf94189ac6b0a0e"}
Oct 30 13:12:05.212294 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:05.212277    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-deb5d83f-9bcf86c6c-87gkt"
Oct 30 13:12:05.214673 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:05.214515    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c" event={"ID":"7709620c-73e4-4a9e-8b06-ef35865ea56d","Type":"ContainerStarted","Data":"7ecc2f9a03aeb4e4c9d7a283e4c56e0a37ec07bd22faf115a220121494399478"}
Oct 30 13:12:05.214673 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:05.214555    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c" event={"ID":"7709620c-73e4-4a9e-8b06-ef35865ea56d","Type":"ContainerStarted","Data":"486b2b6bec8b5202c3147e355f738f6a59988bf5545d3e04a700301bcaeb104e"}
Oct 30 13:12:05.216432 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:05.216402    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-662ed421-7dbbd45746-c5rj6" event={"ID":"2cc2a92b-d546-4cca-a0a0-fd23d881a30b","Type":"ContainerStarted","Data":"e3d8f319077a8fbfe46e51b0ef8eff466d354db0d01e5bdac5180643ebbde6c1"}
Oct 30 13:12:05.216543 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:05.216461    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-662ed421-7dbbd45746-c5rj6"
Oct 30 13:12:05.256619 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:05.256556    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-662ed421-5b6df6999f-b69dp" podStartSLOduration=1.796345694 podStartE2EDuration="5.25653633s" totalImagesPullingTime="3.460190636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:05.252502731 +0000 UTC m=+544.399769998" watchObservedRunningTime="2026-10-30 13:12:05.25653633 +0000 UTC m=+544.403803570"
Oct 30 13:12:05.300124 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:05.300066    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c" podStartSLOduration=2.300050827 podStartE2EDuration="2.300050827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:05.296504507 +0000 UTC m=+544.443771749" watchObservedRunningTime="2026-10-30 13:12:05.300050827 +0000 UTC m=+544.447318068"
Oct 30 13:12:05.352831 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:05.341071    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-deb5d83f-9bcf86c6c-87gkt" podStartSLOduration=2.341049051 podStartE2EDuration="2.341049051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:05.337586403 +0000 UTC m=+544.484853648" watchObservedRunningTime="2026-10-30 13:12:05.341049051 +0000 UTC m=+544.488316294"
Oct 30 13:12:05.379862 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:05.379800    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-662ed421-7dbbd45746-c5rj6" podStartSLOduration=2.195644277 podStartE2EDuration="5.379776891s" totalImagesPullingTime="3.184132614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:05.379187075 +0000 UTC m=+544.526454315" watchObservedRunningTime="2026-10-30 13:12:05.379776891 +0000 UTC m=+544.527044132"
Oct 30 13:12:06.158438 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:06.158403    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c"
Oct 30 13:12:06.164697 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:06.164664    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c"
Oct 30 13:12:06.224022 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:06.223952    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c"
Oct 30 13:12:06.233157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:06.231399    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c"
Oct 30 13:12:07.263472 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:07.263434    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-662ed421-84ddd86fb7-lz7dl" event={"ID":"a54fa560-1acc-4d98-adea-3648e998da77","Type":"ContainerStarted","Data":"f3948c1f7026fbd363f44018737d9446573ad73eedd30b0eff36fa48484403fa"}
Oct 30 13:12:07.264086 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:07.264047    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-662ed421-84ddd86fb7-lz7dl"
Oct 30 13:12:07.306897 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:07.306836    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-662ed421-84ddd86fb7-lz7dl" podStartSLOduration=2.009465418 podStartE2EDuration="7.306822177s" totalImagesPullingTime="5.297356759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:07.305033079 +0000 UTC m=+546.452300331" watchObservedRunningTime="2026-10-30 13:12:07.306822177 +0000 UTC m=+546.454089417"
Oct 30 13:12:09.203616 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:09.203577    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fb3b54cb-8676cfd875-p8rhj"]
Oct 30 13:12:09.207496 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:09.207477    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-fb3b54cb-8676cfd875-p8rhj"
Oct 30 13:12:09.221046 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:09.221019    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-fb3b54cb-7dbc6cbff8-s2dgh"]
Oct 30 13:12:09.225753 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:09.225728    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-fb3b54cb-7dbc6cbff8-s2dgh"
Oct 30 13:12:09.229653 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:09.229625    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fb3b54cb-8676cfd875-p8rhj"]
Oct 30 13:12:09.231775 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:09.231609    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-fb3b54cb\""
Oct 30 13:12:09.244013 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:09.243977    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-fb3b54cb-7dbc6cbff8-s2dgh"]
Oct 30 13:12:09.251203 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:09.251179    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-fb3b54cb-dockercfg-kxw2b\""
Oct 30 13:12:09.289095 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:09.289052    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ntc5\" (UniqueName: \"kubernetes.io/projected/7d13191e-0309-4ff5-87f0-d7366e2c6146-kube-api-access-7ntc5\") pod \"payload-pre-processing-e2e-derive-fb3b54cb-8676cfd875-p8rhj\" (UID: \"7d13191e-0309-4ff5-87f0-d7366e2c6146\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fb3b54cb-8676cfd875-p8rhj"
Oct 30 13:12:09.289250 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:09.289194    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d13191e-0309-4ff5-87f0-d7366e2c6146-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-fb3b54cb-8676cfd875-p8rhj\" (UID: \"7d13191e-0309-4ff5-87f0-d7366e2c6146\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fb3b54cb-8676cfd875-p8rhj"
Oct 30 13:12:09.398176 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:09.389976    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh484\" (UniqueName: \"kubernetes.io/projected/550de091-32fb-4959-8f10-cf3c7b43837f-kube-api-access-mh484\") pod \"payload-processing-e2e-derive-fb3b54cb-7dbc6cbff8-s2dgh\" (UID: \"550de091-32fb-4959-8f10-cf3c7b43837f\") " pod="openshift-ingress/payload-processing-e2e-derive-fb3b54cb-7dbc6cbff8-s2dgh"
Oct 30 13:12:09.398176 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:09.390079    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d13191e-0309-4ff5-87f0-d7366e2c6146-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-fb3b54cb-8676cfd875-p8rhj\" (UID: \"7d13191e-0309-4ff5-87f0-d7366e2c6146\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fb3b54cb-8676cfd875-p8rhj"
Oct 30 13:12:09.398176 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:09.390240    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/550de091-32fb-4959-8f10-cf3c7b43837f-plugins-config-volume\") pod \"payload-processing-e2e-derive-fb3b54cb-7dbc6cbff8-s2dgh\" (UID: \"550de091-32fb-4959-8f10-cf3c7b43837f\") " pod="openshift-ingress/payload-processing-e2e-derive-fb3b54cb-7dbc6cbff8-s2dgh"
Oct 30 13:12:09.398176 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:09.390296    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ntc5\" (UniqueName: \"kubernetes.io/projected/7d13191e-0309-4ff5-87f0-d7366e2c6146-kube-api-access-7ntc5\") pod \"payload-pre-processing-e2e-derive-fb3b54cb-8676cfd875-p8rhj\" (UID: \"7d13191e-0309-4ff5-87f0-d7366e2c6146\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fb3b54cb-8676cfd875-p8rhj"
Oct 30 13:12:09.398176 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:09.391613    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d13191e-0309-4ff5-87f0-d7366e2c6146-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-fb3b54cb-8676cfd875-p8rhj\" (UID: \"7d13191e-0309-4ff5-87f0-d7366e2c6146\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fb3b54cb-8676cfd875-p8rhj"
Oct 30 13:12:09.413900 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:09.413823    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ntc5\" (UniqueName: \"kubernetes.io/projected/7d13191e-0309-4ff5-87f0-d7366e2c6146-kube-api-access-7ntc5\") pod \"payload-pre-processing-e2e-derive-fb3b54cb-8676cfd875-p8rhj\" (UID: \"7d13191e-0309-4ff5-87f0-d7366e2c6146\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fb3b54cb-8676cfd875-p8rhj"
Oct 30 13:12:09.491124 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:09.491019    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/550de091-32fb-4959-8f10-cf3c7b43837f-plugins-config-volume\") pod \"payload-processing-e2e-derive-fb3b54cb-7dbc6cbff8-s2dgh\" (UID: \"550de091-32fb-4959-8f10-cf3c7b43837f\") " pod="openshift-ingress/payload-processing-e2e-derive-fb3b54cb-7dbc6cbff8-s2dgh"
Oct 30 13:12:09.491124 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:09.491093    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh484\" (UniqueName: \"kubernetes.io/projected/550de091-32fb-4959-8f10-cf3c7b43837f-kube-api-access-mh484\") pod \"payload-processing-e2e-derive-fb3b54cb-7dbc6cbff8-s2dgh\" (UID: \"550de091-32fb-4959-8f10-cf3c7b43837f\") " pod="openshift-ingress/payload-processing-e2e-derive-fb3b54cb-7dbc6cbff8-s2dgh"
Oct 30 13:12:09.491978 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:09.491945    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/550de091-32fb-4959-8f10-cf3c7b43837f-plugins-config-volume\") pod \"payload-processing-e2e-derive-fb3b54cb-7dbc6cbff8-s2dgh\" (UID: \"550de091-32fb-4959-8f10-cf3c7b43837f\") " pod="openshift-ingress/payload-processing-e2e-derive-fb3b54cb-7dbc6cbff8-s2dgh"
Oct 30 13:12:09.512673 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:09.512636    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh484\" (UniqueName: \"kubernetes.io/projected/550de091-32fb-4959-8f10-cf3c7b43837f-kube-api-access-mh484\") pod \"payload-processing-e2e-derive-fb3b54cb-7dbc6cbff8-s2dgh\" (UID: \"550de091-32fb-4959-8f10-cf3c7b43837f\") " pod="openshift-ingress/payload-processing-e2e-derive-fb3b54cb-7dbc6cbff8-s2dgh"
Oct 30 13:12:09.524046 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:09.524014    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-fb3b54cb-8676cfd875-p8rhj"
Oct 30 13:12:09.543883 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:09.543845    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-fb3b54cb-7dbc6cbff8-s2dgh"
Oct 30 13:12:09.831886 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:09.831854    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fb3b54cb-8676cfd875-p8rhj"]
Oct 30 13:12:09.834364 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:12:09.834334    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d13191e_0309_4ff5_87f0_d7366e2c6146.slice/crio-14eec9978a35698bf632520ec1909f1a9cf8630940bb09a2ee655afc88b3c4be WatchSource:0}: Error finding container 14eec9978a35698bf632520ec1909f1a9cf8630940bb09a2ee655afc88b3c4be: Status 404 returned error can't find the container with id 14eec9978a35698bf632520ec1909f1a9cf8630940bb09a2ee655afc88b3c4be
Oct 30 13:12:09.874692 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:09.874660    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-fb3b54cb-7dbc6cbff8-s2dgh"]
Oct 30 13:12:09.875932 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:12:09.875899    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod550de091_32fb_4959_8f10_cf3c7b43837f.slice/crio-4f3a8825aeec4d7468573973ea35bf14b71d521e16ce7ad5f32264e32d88393d WatchSource:0}: Error finding container 4f3a8825aeec4d7468573973ea35bf14b71d521e16ce7ad5f32264e32d88393d: Status 404 returned error can't find the container with id 4f3a8825aeec4d7468573973ea35bf14b71d521e16ce7ad5f32264e32d88393d
Oct 30 13:12:10.287917 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:10.287875    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-fb3b54cb-8676cfd875-p8rhj" event={"ID":"7d13191e-0309-4ff5-87f0-d7366e2c6146","Type":"ContainerStarted","Data":"0d8564ad876329b389db8a79c3d44b361b7dd04ca405b9939d18c22ad48387d6"}
Oct 30 13:12:10.288450 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:10.287942    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-fb3b54cb-8676cfd875-p8rhj" event={"ID":"7d13191e-0309-4ff5-87f0-d7366e2c6146","Type":"ContainerStarted","Data":"14eec9978a35698bf632520ec1909f1a9cf8630940bb09a2ee655afc88b3c4be"}
Oct 30 13:12:10.288450 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:10.287969    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-fb3b54cb-8676cfd875-p8rhj"
Oct 30 13:12:10.290193 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:10.290160    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-fb3b54cb-7dbc6cbff8-s2dgh" event={"ID":"550de091-32fb-4959-8f10-cf3c7b43837f","Type":"ContainerStarted","Data":"e7864c3009c65556a3816b7d54a6c5805d0d2fd2c9cc2b87c4fdbb969076c735"}
Oct 30 13:12:10.290330 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:10.290216    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-fb3b54cb-7dbc6cbff8-s2dgh" event={"ID":"550de091-32fb-4959-8f10-cf3c7b43837f","Type":"ContainerStarted","Data":"4f3a8825aeec4d7468573973ea35bf14b71d521e16ce7ad5f32264e32d88393d"}
Oct 30 13:12:10.290330 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:10.290238    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-fb3b54cb-7dbc6cbff8-s2dgh"
Oct 30 13:12:10.331365 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:10.331291    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-fb3b54cb-8676cfd875-p8rhj" podStartSLOduration=1.331269702 podStartE2EDuration="1.331269702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:10.328046033 +0000 UTC m=+549.475313290" watchObservedRunningTime="2026-10-30 13:12:10.331269702 +0000 UTC m=+549.478536945"
Oct 30 13:12:10.368757 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:10.368701    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-fb3b54cb-7dbc6cbff8-s2dgh" podStartSLOduration=1.368685606 podStartE2EDuration="1.368685606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:10.367728505 +0000 UTC m=+549.514995754" watchObservedRunningTime="2026-10-30 13:12:10.368685606 +0000 UTC m=+549.515952839"
Oct 30 13:12:12.024363 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:12:12.024327    5093 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6babd5ad_925c_4ac6_ab8a_9dffbb090259.slice/crio-conmon-01b42cc629afb4f4db97ef5a7f15af1bbc2ff77afb578ec3dae5cda8d0e2dc89.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 13:12:12.024849 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:12:12.024388    5093 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6babd5ad_925c_4ac6_ab8a_9dffbb090259.slice/crio-conmon-01b42cc629afb4f4db97ef5a7f15af1bbc2ff77afb578ec3dae5cda8d0e2dc89.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 13:12:12.057213 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:12.057178    5093 patch_prober.go:29] interesting pod/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 30 13:12:12.057399 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:12.057235    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd" podUID="6babd5ad-925c-4ac6-ab8a-9dffbb090259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 30 13:12:12.308593 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:12.308490    5093 generic.go:356] "Generic (PLEG): container finished" podID="6babd5ad-925c-4ac6-ab8a-9dffbb090259" containerID="01b42cc629afb4f4db97ef5a7f15af1bbc2ff77afb578ec3dae5cda8d0e2dc89" exitCode=0
Oct 30 13:12:12.308593 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:12.308554    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd" event={"ID":"6babd5ad-925c-4ac6-ab8a-9dffbb090259","Type":"ContainerDied","Data":"01b42cc629afb4f4db97ef5a7f15af1bbc2ff77afb578ec3dae5cda8d0e2dc89"}
Oct 30 13:12:12.309358 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:12.309334    5093 scope.go:122] "RemoveContainer" containerID="01b42cc629afb4f4db97ef5a7f15af1bbc2ff77afb578ec3dae5cda8d0e2dc89"
Oct 30 13:12:12.311086 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:12.311039    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:12:12.311236 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:12.311161    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:12:12.311236 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:12.311215    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:12:13.277659 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:13.277631    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-662ed421-84ddd86fb7-lz7dl"
Oct 30 13:12:13.317769 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:13.317737    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd" event={"ID":"6babd5ad-925c-4ac6-ab8a-9dffbb090259","Type":"ContainerStarted","Data":"e3308aa451ca9cc6ebab75d938421a7fd9888a1ffc5535b710f07fe08d1afd54"}
Oct 30 13:12:14.081232 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:14.081194    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd"
Oct 30 13:12:14.081481 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:14.081459    5093 patch_prober.go:29] interesting pod/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 30 13:12:14.081536 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:14.081508    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd" podUID="6babd5ad-925c-4ac6-ab8a-9dffbb090259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 30 13:12:15.081564 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:15.081531    5093 patch_prober.go:29] interesting pod/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 30 13:12:15.081977 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:15.081591    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd" podUID="6babd5ad-925c-4ac6-ab8a-9dffbb090259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 30 13:12:15.219981 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:15.219948    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d6a10701-84f4cccb8c-k4552"
Oct 30 13:12:16.081210 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:16.081115    5093 patch_prober.go:29] interesting pod/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 30 13:12:16.081427 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:16.081227    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd" podUID="6babd5ad-925c-4ac6-ab8a-9dffbb090259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 30 13:12:16.254157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:16.254081    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-deb5d83f-9bcf86c6c-87gkt"
Oct 30 13:12:16.255110 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:16.254880    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc4197ef-5f7fb55b75-mps2d"
Oct 30 13:12:16.255110 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:16.254984    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-662ed421-5b6df6999f-b69dp"
Oct 30 13:12:16.256457 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:16.255929    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-662ed421-7dbbd45746-c5rj6"
Oct 30 13:12:17.081378 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:17.081341    5093 patch_prober.go:29] interesting pod/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 30 13:12:17.081568 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:17.081413    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd" podUID="6babd5ad-925c-4ac6-ab8a-9dffbb090259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 30 13:12:18.080832 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:18.080790    5093 patch_prober.go:29] interesting pod/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 30 13:12:18.081264 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:18.080866    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd" podUID="6babd5ad-925c-4ac6-ab8a-9dffbb090259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 30 13:12:19.085664 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:19.085633    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd"
Oct 30 13:12:19.086125 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:19.086112    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd"
Oct 30 13:12:19.087009 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:19.086984    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd"
Oct 30 13:12:21.173682 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:12:21.173638    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:12:21.299372 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:21.299341    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-fb3b54cb-8676cfd875-p8rhj"
Oct 30 13:12:21.299538 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:21.299508    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-fb3b54cb-7dbc6cbff8-s2dgh"
Oct 30 13:12:23.455478 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:23.455440    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d6a10701-84f4cccb8c-k4552"]
Oct 30 13:12:28.456525 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:28.456447    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d6a10701-84f4cccb8c-k4552" podUID="f4d7901b-2969-4fb4-9805-f87f7f2a82b1" containerName="payload-pre-processing" containerID="cri-o://0b235bcaf12a9776f4d223281af5ae366f76270c37d969bc5a5a6f908a9803a0" gracePeriod=25
Oct 30 13:12:28.768611 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:28.768583    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d6a10701-84f4cccb8c-k4552"
Oct 30 13:12:28.839703 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:28.839664    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4d7901b-2969-4fb4-9805-f87f7f2a82b1-kube-api-access-fq856\" (UniqueName: \"kubernetes.io/projected/f4d7901b-2969-4fb4-9805-f87f7f2a82b1-kube-api-access-fq856\") pod \"f4d7901b-2969-4fb4-9805-f87f7f2a82b1\" (UID: \"f4d7901b-2969-4fb4-9805-f87f7f2a82b1\") "
Oct 30 13:12:28.839857 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:28.839807    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4d7901b-2969-4fb4-9805-f87f7f2a82b1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4d7901b-2969-4fb4-9805-f87f7f2a82b1-plugins-config-volume\") pod \"f4d7901b-2969-4fb4-9805-f87f7f2a82b1\" (UID: \"f4d7901b-2969-4fb4-9805-f87f7f2a82b1\") "
Oct 30 13:12:28.840262 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:28.840236    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4d7901b-2969-4fb4-9805-f87f7f2a82b1-plugins-config-volume" pod "f4d7901b-2969-4fb4-9805-f87f7f2a82b1" (UID: "f4d7901b-2969-4fb4-9805-f87f7f2a82b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:12:28.841945 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:28.841913    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4d7901b-2969-4fb4-9805-f87f7f2a82b1-kube-api-access-fq856" pod "f4d7901b-2969-4fb4-9805-f87f7f2a82b1" (UID: "f4d7901b-2969-4fb4-9805-f87f7f2a82b1"). InnerVolumeSpecName "kube-api-access-fq856". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:12:28.941678 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:28.941639    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq856\" (UniqueName: \"kubernetes.io/projected/f4d7901b-2969-4fb4-9805-f87f7f2a82b1-kube-api-access-fq856\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:12:28.941678 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:28.941673    5093 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4d7901b-2969-4fb4-9805-f87f7f2a82b1-plugins-config-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:12:28.983448 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:28.983357    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fc4197ef-5f7fb55b75-mps2d"]
Oct 30 13:12:29.498234 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:29.498195    5093 generic.go:356] "Generic (PLEG): container finished" podID="f4d7901b-2969-4fb4-9805-f87f7f2a82b1" containerID="0b235bcaf12a9776f4d223281af5ae366f76270c37d969bc5a5a6f908a9803a0" exitCode=0
Oct 30 13:12:29.498738 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:29.498249    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d6a10701-84f4cccb8c-k4552"
Oct 30 13:12:29.498738 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:29.498284    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d6a10701-84f4cccb8c-k4552" event={"ID":"f4d7901b-2969-4fb4-9805-f87f7f2a82b1","Type":"ContainerDied","Data":"0b235bcaf12a9776f4d223281af5ae366f76270c37d969bc5a5a6f908a9803a0"}
Oct 30 13:12:29.498738 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:29.498363    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d6a10701-84f4cccb8c-k4552" event={"ID":"f4d7901b-2969-4fb4-9805-f87f7f2a82b1","Type":"ContainerDied","Data":"52d11bf9b3a99e4b66436accfd4c822f5f450e6174c23c66b5ec21e10ce8bed3"}
Oct 30 13:12:29.498738 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:29.498399    5093 scope.go:122] "RemoveContainer" containerID="0b235bcaf12a9776f4d223281af5ae366f76270c37d969bc5a5a6f908a9803a0"
Oct 30 13:12:29.508477 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:29.508453    5093 scope.go:122] "RemoveContainer" containerID="0b235bcaf12a9776f4d223281af5ae366f76270c37d969bc5a5a6f908a9803a0"
Oct 30 13:12:29.508817 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:12:29.508792    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b235bcaf12a9776f4d223281af5ae366f76270c37d969bc5a5a6f908a9803a0\": container with ID starting with 0b235bcaf12a9776f4d223281af5ae366f76270c37d969bc5a5a6f908a9803a0 not found: ID does not exist" containerID="0b235bcaf12a9776f4d223281af5ae366f76270c37d969bc5a5a6f908a9803a0"
Oct 30 13:12:29.508891 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:29.508832    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b235bcaf12a9776f4d223281af5ae366f76270c37d969bc5a5a6f908a9803a0"} err="failed to get container status \"0b235bcaf12a9776f4d223281af5ae366f76270c37d969bc5a5a6f908a9803a0\": rpc error: code = NotFound desc = could not find container \"0b235bcaf12a9776f4d223281af5ae366f76270c37d969bc5a5a6f908a9803a0\": container with ID starting with 0b235bcaf12a9776f4d223281af5ae366f76270c37d969bc5a5a6f908a9803a0 not found: ID does not exist"
Oct 30 13:12:29.553078 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:29.553043    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d6a10701-84f4cccb8c-k4552"]
Oct 30 13:12:29.567075 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:29.567031    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d6a10701-84f4cccb8c-k4552"]
Oct 30 13:12:30.506589 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:30.506555    5093 generic.go:356] "Generic (PLEG): container finished" podID="7709620c-73e4-4a9e-8b06-ef35865ea56d" containerID="7ecc2f9a03aeb4e4c9d7a283e4c56e0a37ec07bd22faf115a220121494399478" exitCode=0
Oct 30 13:12:30.507096 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:30.506628    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c" event={"ID":"7709620c-73e4-4a9e-8b06-ef35865ea56d","Type":"ContainerDied","Data":"7ecc2f9a03aeb4e4c9d7a283e4c56e0a37ec07bd22faf115a220121494399478"}
Oct 30 13:12:30.507096 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:30.507084    5093 scope.go:122] "RemoveContainer" containerID="7ecc2f9a03aeb4e4c9d7a283e4c56e0a37ec07bd22faf115a220121494399478"
Oct 30 13:12:30.508506 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:30.508476    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:12:30.508605 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:30.508566    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:12:30.508605 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:30.508599    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:12:30.998158 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:30.998112    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4d7901b-2969-4fb4-9805-f87f7f2a82b1" path="/var/lib/kubelet/pods/f4d7901b-2969-4fb4-9805-f87f7f2a82b1/volumes"
Oct 30 13:12:31.515519 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:31.515485    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c" event={"ID":"7709620c-73e4-4a9e-8b06-ef35865ea56d","Type":"ContainerStarted","Data":"b8d5a6476076174acb54dccff56db985ee563c5afd1911fb4a445e4e6b7c9930"}
Oct 30 13:12:32.158581 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:32.158546    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c"
Oct 30 13:12:32.159412 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:32.159384    5093 patch_prober.go:29] interesting pod/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Oct 30 13:12:32.159532 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:32.159448    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c" podUID="7709620c-73e4-4a9e-8b06-ef35865ea56d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Oct 30 13:12:33.158676 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:33.158644    5093 patch_prober.go:29] interesting pod/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Oct 30 13:12:33.159118 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:33.158702    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c" podUID="7709620c-73e4-4a9e-8b06-ef35865ea56d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Oct 30 13:12:33.766051 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:33.766009    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd"]
Oct 30 13:12:33.766618 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:33.766581    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd" podUID="6babd5ad-925c-4ac6-ab8a-9dffbb090259" containerName="istio-proxy" containerID="cri-o://e3308aa451ca9cc6ebab75d938421a7fd9888a1ffc5535b710f07fe08d1afd54" gracePeriod=30
Oct 30 13:12:33.983727 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:33.983681    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc4197ef-5f7fb55b75-mps2d" podUID="3e9eef72-38bd-4fea-bd12-36d611a3c933" containerName="payload-pre-processing" containerID="cri-o://5486bf167d158788f59105dd5d6d4e1e73a7842e7e87593e96e7e17692268b4b" gracePeriod=25
Oct 30 13:12:34.089484 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:34.089392    5093 patch_prober.go:29] interesting pod/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:12:34.089484 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:34.089461    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd" podUID="6babd5ad-925c-4ac6-ab8a-9dffbb090259" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:12:34.158727 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:34.158686    5093 patch_prober.go:29] interesting pod/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Oct 30 13:12:34.159141 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:34.158746    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c" podUID="7709620c-73e4-4a9e-8b06-ef35865ea56d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Oct 30 13:12:34.298667 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:34.298638    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc4197ef-5f7fb55b75-mps2d"
Oct 30 13:12:34.414203 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:34.414102    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e9eef72-38bd-4fea-bd12-36d611a3c933-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e9eef72-38bd-4fea-bd12-36d611a3c933-plugins-config-volume\") pod \"3e9eef72-38bd-4fea-bd12-36d611a3c933\" (UID: \"3e9eef72-38bd-4fea-bd12-36d611a3c933\") "
Oct 30 13:12:34.414203 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:34.414175    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e9eef72-38bd-4fea-bd12-36d611a3c933-kube-api-access-tzr8b\" (UniqueName: \"kubernetes.io/projected/3e9eef72-38bd-4fea-bd12-36d611a3c933-kube-api-access-tzr8b\") pod \"3e9eef72-38bd-4fea-bd12-36d611a3c933\" (UID: \"3e9eef72-38bd-4fea-bd12-36d611a3c933\") "
Oct 30 13:12:34.414574 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:34.414546    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e9eef72-38bd-4fea-bd12-36d611a3c933-plugins-config-volume" pod "3e9eef72-38bd-4fea-bd12-36d611a3c933" (UID: "3e9eef72-38bd-4fea-bd12-36d611a3c933"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:12:34.416283 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:34.416259    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e9eef72-38bd-4fea-bd12-36d611a3c933-kube-api-access-tzr8b" pod "3e9eef72-38bd-4fea-bd12-36d611a3c933" (UID: "3e9eef72-38bd-4fea-bd12-36d611a3c933"). InnerVolumeSpecName "kube-api-access-tzr8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:12:34.515273 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:34.515239    5093 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e9eef72-38bd-4fea-bd12-36d611a3c933-plugins-config-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:12:34.515273 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:34.515269    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzr8b\" (UniqueName: \"kubernetes.io/projected/3e9eef72-38bd-4fea-bd12-36d611a3c933-kube-api-access-tzr8b\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:12:34.558480 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:34.558444    5093 generic.go:356] "Generic (PLEG): container finished" podID="3e9eef72-38bd-4fea-bd12-36d611a3c933" containerID="5486bf167d158788f59105dd5d6d4e1e73a7842e7e87593e96e7e17692268b4b" exitCode=0
Oct 30 13:12:34.558683 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:34.558518    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc4197ef-5f7fb55b75-mps2d"
Oct 30 13:12:34.558683 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:34.558528    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc4197ef-5f7fb55b75-mps2d" event={"ID":"3e9eef72-38bd-4fea-bd12-36d611a3c933","Type":"ContainerDied","Data":"5486bf167d158788f59105dd5d6d4e1e73a7842e7e87593e96e7e17692268b4b"}
Oct 30 13:12:34.558683 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:34.558583    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc4197ef-5f7fb55b75-mps2d" event={"ID":"3e9eef72-38bd-4fea-bd12-36d611a3c933","Type":"ContainerDied","Data":"a1c0da2f0547b54d3536dca8a972e4de601bc66c61ef7b0ad5cdf887f522d5a2"}
Oct 30 13:12:34.558683 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:34.558607    5093 scope.go:122] "RemoveContainer" containerID="5486bf167d158788f59105dd5d6d4e1e73a7842e7e87593e96e7e17692268b4b"
Oct 30 13:12:34.568571 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:34.568550    5093 scope.go:122] "RemoveContainer" containerID="5486bf167d158788f59105dd5d6d4e1e73a7842e7e87593e96e7e17692268b4b"
Oct 30 13:12:34.568879 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:12:34.568855    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5486bf167d158788f59105dd5d6d4e1e73a7842e7e87593e96e7e17692268b4b\": container with ID starting with 5486bf167d158788f59105dd5d6d4e1e73a7842e7e87593e96e7e17692268b4b not found: ID does not exist" containerID="5486bf167d158788f59105dd5d6d4e1e73a7842e7e87593e96e7e17692268b4b"
Oct 30 13:12:34.569000 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:34.568886    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5486bf167d158788f59105dd5d6d4e1e73a7842e7e87593e96e7e17692268b4b"} err="failed to get container status \"5486bf167d158788f59105dd5d6d4e1e73a7842e7e87593e96e7e17692268b4b\": rpc error: code = NotFound desc = could not find container \"5486bf167d158788f59105dd5d6d4e1e73a7842e7e87593e96e7e17692268b4b\": container with ID starting with 5486bf167d158788f59105dd5d6d4e1e73a7842e7e87593e96e7e17692268b4b not found: ID does not exist"
Oct 30 13:12:34.614423 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:34.614384    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fc4197ef-5f7fb55b75-mps2d"]
Oct 30 13:12:34.637979 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:34.637901    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fc4197ef-5f7fb55b75-mps2d"]
Oct 30 13:12:34.999791 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:34.999760    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e9eef72-38bd-4fea-bd12-36d611a3c933" path="/var/lib/kubelet/pods/3e9eef72-38bd-4fea-bd12-36d611a3c933/volumes"
Oct 30 13:12:35.159154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:35.158949    5093 patch_prober.go:29] interesting pod/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Oct 30 13:12:35.159154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:35.159017    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c" podUID="7709620c-73e4-4a9e-8b06-ef35865ea56d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Oct 30 13:12:36.168306 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:36.168260    5093 patch_prober.go:29] interesting pod/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Oct 30 13:12:36.185181 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:36.178252    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c" podUID="7709620c-73e4-4a9e-8b06-ef35865ea56d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Oct 30 13:12:36.223894 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:36.223845    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c"
Oct 30 13:12:36.538320 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:36.538283    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-deb5d83f-9bcf86c6c-87gkt"]
Oct 30 13:12:37.162698 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:37.162663    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c"
Oct 30 13:12:37.163935 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:37.163911    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c"
Oct 30 13:12:39.038479 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:39.038418    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd"
Oct 30 13:12:39.207275 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:39.207244    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6babd5ad-925c-4ac6-ab8a-9dffbb090259-kube-api-access-wvmw9\" (UniqueName: \"kubernetes.io/projected/6babd5ad-925c-4ac6-ab8a-9dffbb090259-kube-api-access-wvmw9\") pod \"6babd5ad-925c-4ac6-ab8a-9dffbb090259\" (UID: \"6babd5ad-925c-4ac6-ab8a-9dffbb090259\") "
Oct 30 13:12:39.207441 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:39.207312    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6babd5ad-925c-4ac6-ab8a-9dffbb090259-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6babd5ad-925c-4ac6-ab8a-9dffbb090259-workload-socket\") pod \"6babd5ad-925c-4ac6-ab8a-9dffbb090259\" (UID: \"6babd5ad-925c-4ac6-ab8a-9dffbb090259\") "
Oct 30 13:12:39.207441 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:39.207340    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6babd5ad-925c-4ac6-ab8a-9dffbb090259-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6babd5ad-925c-4ac6-ab8a-9dffbb090259-istio-envoy\") pod \"6babd5ad-925c-4ac6-ab8a-9dffbb090259\" (UID: \"6babd5ad-925c-4ac6-ab8a-9dffbb090259\") "
Oct 30 13:12:39.207441 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:39.207405    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6babd5ad-925c-4ac6-ab8a-9dffbb090259-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6babd5ad-925c-4ac6-ab8a-9dffbb090259-istio-podinfo\") pod \"6babd5ad-925c-4ac6-ab8a-9dffbb090259\" (UID: \"6babd5ad-925c-4ac6-ab8a-9dffbb090259\") "
Oct 30 13:12:39.207550 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:39.207451    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6babd5ad-925c-4ac6-ab8a-9dffbb090259-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6babd5ad-925c-4ac6-ab8a-9dffbb090259-istio-data\") pod \"6babd5ad-925c-4ac6-ab8a-9dffbb090259\" (UID: \"6babd5ad-925c-4ac6-ab8a-9dffbb090259\") "
Oct 30 13:12:39.207659 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:39.207614    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6babd5ad-925c-4ac6-ab8a-9dffbb090259-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6babd5ad-925c-4ac6-ab8a-9dffbb090259-workload-certs\") pod \"6babd5ad-925c-4ac6-ab8a-9dffbb090259\" (UID: \"6babd5ad-925c-4ac6-ab8a-9dffbb090259\") "
Oct 30 13:12:39.207721 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:39.207664    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6babd5ad-925c-4ac6-ab8a-9dffbb090259-workload-socket" pod "6babd5ad-925c-4ac6-ab8a-9dffbb090259" (UID: "6babd5ad-925c-4ac6-ab8a-9dffbb090259"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:12:39.207721 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:39.207695    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6babd5ad-925c-4ac6-ab8a-9dffbb090259-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6babd5ad-925c-4ac6-ab8a-9dffbb090259-istiod-ca-cert\") pod \"6babd5ad-925c-4ac6-ab8a-9dffbb090259\" (UID: \"6babd5ad-925c-4ac6-ab8a-9dffbb090259\") "
Oct 30 13:12:39.207829 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:39.207741    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6babd5ad-925c-4ac6-ab8a-9dffbb090259-istio-data" pod "6babd5ad-925c-4ac6-ab8a-9dffbb090259" (UID: "6babd5ad-925c-4ac6-ab8a-9dffbb090259"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:12:39.207829 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:39.207779    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6babd5ad-925c-4ac6-ab8a-9dffbb090259-istio-token\" (UniqueName: \"kubernetes.io/projected/6babd5ad-925c-4ac6-ab8a-9dffbb090259-istio-token\") pod \"6babd5ad-925c-4ac6-ab8a-9dffbb090259\" (UID: \"6babd5ad-925c-4ac6-ab8a-9dffbb090259\") "
Oct 30 13:12:39.207942 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:39.207848    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6babd5ad-925c-4ac6-ab8a-9dffbb090259-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6babd5ad-925c-4ac6-ab8a-9dffbb090259-credential-socket\") pod \"6babd5ad-925c-4ac6-ab8a-9dffbb090259\" (UID: \"6babd5ad-925c-4ac6-ab8a-9dffbb090259\") "
Oct 30 13:12:39.207999 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:39.207931    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6babd5ad-925c-4ac6-ab8a-9dffbb090259-workload-certs" pod "6babd5ad-925c-4ac6-ab8a-9dffbb090259" (UID: "6babd5ad-925c-4ac6-ab8a-9dffbb090259"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:12:39.208054 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:39.208006    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6babd5ad-925c-4ac6-ab8a-9dffbb090259-istiod-ca-cert" pod "6babd5ad-925c-4ac6-ab8a-9dffbb090259" (UID: "6babd5ad-925c-4ac6-ab8a-9dffbb090259"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:12:39.208242 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:39.208212    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6babd5ad-925c-4ac6-ab8a-9dffbb090259-credential-socket" pod "6babd5ad-925c-4ac6-ab8a-9dffbb090259" (UID: "6babd5ad-925c-4ac6-ab8a-9dffbb090259"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:12:39.208394 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:39.208371    5093 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6babd5ad-925c-4ac6-ab8a-9dffbb090259-istio-data\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:12:39.208473 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:39.208398    5093 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6babd5ad-925c-4ac6-ab8a-9dffbb090259-workload-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:12:39.208473 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:39.208414    5093 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6babd5ad-925c-4ac6-ab8a-9dffbb090259-istiod-ca-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:12:39.208473 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:39.208427    5093 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6babd5ad-925c-4ac6-ab8a-9dffbb090259-credential-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:12:39.208473 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:39.208444    5093 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6babd5ad-925c-4ac6-ab8a-9dffbb090259-workload-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:12:39.209724 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:39.209698    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6babd5ad-925c-4ac6-ab8a-9dffbb090259-istio-podinfo" pod "6babd5ad-925c-4ac6-ab8a-9dffbb090259" (UID: "6babd5ad-925c-4ac6-ab8a-9dffbb090259"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:12:39.209824 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:39.209745    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6babd5ad-925c-4ac6-ab8a-9dffbb090259-istio-envoy" pod "6babd5ad-925c-4ac6-ab8a-9dffbb090259" (UID: "6babd5ad-925c-4ac6-ab8a-9dffbb090259"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:12:39.209824 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:39.209774    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6babd5ad-925c-4ac6-ab8a-9dffbb090259-kube-api-access-wvmw9" pod "6babd5ad-925c-4ac6-ab8a-9dffbb090259" (UID: "6babd5ad-925c-4ac6-ab8a-9dffbb090259"). InnerVolumeSpecName "kube-api-access-wvmw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:12:39.209904 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:39.209839    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6babd5ad-925c-4ac6-ab8a-9dffbb090259-istio-token" pod "6babd5ad-925c-4ac6-ab8a-9dffbb090259" (UID: "6babd5ad-925c-4ac6-ab8a-9dffbb090259"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:12:39.278224 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:39.278181    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-662ed421-7dbbd45746-c5rj6"]
Oct 30 13:12:39.307088 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:39.306988    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-662ed421-5b6df6999f-b69dp"]
Oct 30 13:12:39.309674 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:39.309642    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6babd5ad-925c-4ac6-ab8a-9dffbb090259-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:12:39.309674 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:39.309672    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvmw9\" (UniqueName: \"kubernetes.io/projected/6babd5ad-925c-4ac6-ab8a-9dffbb090259-kube-api-access-wvmw9\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:12:39.309848 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:39.309686    5093 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6babd5ad-925c-4ac6-ab8a-9dffbb090259-istio-envoy\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:12:39.309848 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:39.309701    5093 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6babd5ad-925c-4ac6-ab8a-9dffbb090259-istio-podinfo\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:12:39.596914 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:39.596828    5093 generic.go:356] "Generic (PLEG): container finished" podID="6babd5ad-925c-4ac6-ab8a-9dffbb090259" containerID="e3308aa451ca9cc6ebab75d938421a7fd9888a1ffc5535b710f07fe08d1afd54" exitCode=0
Oct 30 13:12:39.596914 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:39.596883    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd"
Oct 30 13:12:39.597094 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:39.596911    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd" event={"ID":"6babd5ad-925c-4ac6-ab8a-9dffbb090259","Type":"ContainerDied","Data":"e3308aa451ca9cc6ebab75d938421a7fd9888a1ffc5535b710f07fe08d1afd54"}
Oct 30 13:12:39.597094 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:39.596970    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd" event={"ID":"6babd5ad-925c-4ac6-ab8a-9dffbb090259","Type":"ContainerDied","Data":"7967b8b2fd86b50618392e60502f8c718215928cd8634d42ef136cc80f1bd1f7"}
Oct 30 13:12:39.597094 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:39.596994    5093 scope.go:122] "RemoveContainer" containerID="e3308aa451ca9cc6ebab75d938421a7fd9888a1ffc5535b710f07fe08d1afd54"
Oct 30 13:12:39.609787 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:39.609765    5093 scope.go:122] "RemoveContainer" containerID="01b42cc629afb4f4db97ef5a7f15af1bbc2ff77afb578ec3dae5cda8d0e2dc89"
Oct 30 13:12:39.619394 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:39.619371    5093 scope.go:122] "RemoveContainer" containerID="e3308aa451ca9cc6ebab75d938421a7fd9888a1ffc5535b710f07fe08d1afd54"
Oct 30 13:12:39.619700 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:12:39.619679    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3308aa451ca9cc6ebab75d938421a7fd9888a1ffc5535b710f07fe08d1afd54\": container with ID starting with e3308aa451ca9cc6ebab75d938421a7fd9888a1ffc5535b710f07fe08d1afd54 not found: ID does not exist" containerID="e3308aa451ca9cc6ebab75d938421a7fd9888a1ffc5535b710f07fe08d1afd54"
Oct 30 13:12:39.619749 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:39.619710    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3308aa451ca9cc6ebab75d938421a7fd9888a1ffc5535b710f07fe08d1afd54"} err="failed to get container status \"e3308aa451ca9cc6ebab75d938421a7fd9888a1ffc5535b710f07fe08d1afd54\": rpc error: code = NotFound desc = could not find container \"e3308aa451ca9cc6ebab75d938421a7fd9888a1ffc5535b710f07fe08d1afd54\": container with ID starting with e3308aa451ca9cc6ebab75d938421a7fd9888a1ffc5535b710f07fe08d1afd54 not found: ID does not exist"
Oct 30 13:12:39.619749 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:39.619729    5093 scope.go:122] "RemoveContainer" containerID="01b42cc629afb4f4db97ef5a7f15af1bbc2ff77afb578ec3dae5cda8d0e2dc89"
Oct 30 13:12:39.619981 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:12:39.619963    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01b42cc629afb4f4db97ef5a7f15af1bbc2ff77afb578ec3dae5cda8d0e2dc89\": container with ID starting with 01b42cc629afb4f4db97ef5a7f15af1bbc2ff77afb578ec3dae5cda8d0e2dc89 not found: ID does not exist" containerID="01b42cc629afb4f4db97ef5a7f15af1bbc2ff77afb578ec3dae5cda8d0e2dc89"
Oct 30 13:12:39.620029 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:39.619988    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01b42cc629afb4f4db97ef5a7f15af1bbc2ff77afb578ec3dae5cda8d0e2dc89"} err="failed to get container status \"01b42cc629afb4f4db97ef5a7f15af1bbc2ff77afb578ec3dae5cda8d0e2dc89\": rpc error: code = NotFound desc = could not find container \"01b42cc629afb4f4db97ef5a7f15af1bbc2ff77afb578ec3dae5cda8d0e2dc89\": container with ID starting with 01b42cc629afb4f4db97ef5a7f15af1bbc2ff77afb578ec3dae5cda8d0e2dc89 not found: ID does not exist"
Oct 30 13:12:39.653676 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:39.653635    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd"]
Oct 30 13:12:39.657151 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:39.657114    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-d6a10701-gw-openshift-default-6d6698568f-p6kwd"]
Oct 30 13:12:39.892196 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:39.892077    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-662ed421-84ddd86fb7-lz7dl"]
Oct 30 13:12:39.892369 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:39.892312    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-662ed421-84ddd86fb7-lz7dl" podUID="a54fa560-1acc-4d98-adea-3648e998da77" containerName="maas-api" containerID="cri-o://f3948c1f7026fbd363f44018737d9446573ad73eedd30b0eff36fa48484403fa" gracePeriod=30
Oct 30 13:12:40.165864 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:40.165784    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-662ed421-84ddd86fb7-lz7dl"
Oct 30 13:12:40.319753 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:40.319717    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a54fa560-1acc-4d98-adea-3648e998da77-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a54fa560-1acc-4d98-adea-3648e998da77-openshift-service-ca\") pod \"a54fa560-1acc-4d98-adea-3648e998da77\" (UID: \"a54fa560-1acc-4d98-adea-3648e998da77\") "
Oct 30 13:12:40.319936 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:40.319781    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a54fa560-1acc-4d98-adea-3648e998da77-kube-api-access-rtb26\" (UniqueName: \"kubernetes.io/projected/a54fa560-1acc-4d98-adea-3648e998da77-kube-api-access-rtb26\") pod \"a54fa560-1acc-4d98-adea-3648e998da77\" (UID: \"a54fa560-1acc-4d98-adea-3648e998da77\") "
Oct 30 13:12:40.319936 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:40.319815    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a54fa560-1acc-4d98-adea-3648e998da77-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a54fa560-1acc-4d98-adea-3648e998da77-metrics-certs\") pod \"a54fa560-1acc-4d98-adea-3648e998da77\" (UID: \"a54fa560-1acc-4d98-adea-3648e998da77\") "
Oct 30 13:12:40.319936 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:40.319844    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a54fa560-1acc-4d98-adea-3648e998da77-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a54fa560-1acc-4d98-adea-3648e998da77-maas-api-tls\") pod \"a54fa560-1acc-4d98-adea-3648e998da77\" (UID: \"a54fa560-1acc-4d98-adea-3648e998da77\") "
Oct 30 13:12:40.320251 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:40.320218    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a54fa560-1acc-4d98-adea-3648e998da77-openshift-service-ca" pod "a54fa560-1acc-4d98-adea-3648e998da77" (UID: "a54fa560-1acc-4d98-adea-3648e998da77"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:12:40.322215 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:40.322188    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a54fa560-1acc-4d98-adea-3648e998da77-metrics-certs" pod "a54fa560-1acc-4d98-adea-3648e998da77" (UID: "a54fa560-1acc-4d98-adea-3648e998da77"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:12:40.322494 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:40.322465    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a54fa560-1acc-4d98-adea-3648e998da77-kube-api-access-rtb26" pod "a54fa560-1acc-4d98-adea-3648e998da77" (UID: "a54fa560-1acc-4d98-adea-3648e998da77"). InnerVolumeSpecName "kube-api-access-rtb26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:12:40.322583 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:40.322522    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a54fa560-1acc-4d98-adea-3648e998da77-maas-api-tls" pod "a54fa560-1acc-4d98-adea-3648e998da77" (UID: "a54fa560-1acc-4d98-adea-3648e998da77"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:12:40.420866 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:40.420770    5093 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a54fa560-1acc-4d98-adea-3648e998da77-openshift-service-ca\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:12:40.420866 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:40.420802    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtb26\" (UniqueName: \"kubernetes.io/projected/a54fa560-1acc-4d98-adea-3648e998da77-kube-api-access-rtb26\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:12:40.420866 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:40.420812    5093 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a54fa560-1acc-4d98-adea-3648e998da77-metrics-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:12:40.420866 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:40.420821    5093 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a54fa560-1acc-4d98-adea-3648e998da77-maas-api-tls\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:12:40.604580 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:40.604545    5093 generic.go:356] "Generic (PLEG): container finished" podID="a54fa560-1acc-4d98-adea-3648e998da77" containerID="f3948c1f7026fbd363f44018737d9446573ad73eedd30b0eff36fa48484403fa" exitCode=0
Oct 30 13:12:40.604763 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:40.604595    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-662ed421-84ddd86fb7-lz7dl"
Oct 30 13:12:40.604763 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:40.604624    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-662ed421-84ddd86fb7-lz7dl" event={"ID":"a54fa560-1acc-4d98-adea-3648e998da77","Type":"ContainerDied","Data":"f3948c1f7026fbd363f44018737d9446573ad73eedd30b0eff36fa48484403fa"}
Oct 30 13:12:40.604763 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:40.604683    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-662ed421-84ddd86fb7-lz7dl" event={"ID":"a54fa560-1acc-4d98-adea-3648e998da77","Type":"ContainerDied","Data":"427ba00fa0d77e3958e709e6752e507e3cf449e1ad274d5c23689b5c066a81c0"}
Oct 30 13:12:40.604763 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:40.604712    5093 scope.go:122] "RemoveContainer" containerID="f3948c1f7026fbd363f44018737d9446573ad73eedd30b0eff36fa48484403fa"
Oct 30 13:12:40.615325 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:40.615307    5093 scope.go:122] "RemoveContainer" containerID="f3948c1f7026fbd363f44018737d9446573ad73eedd30b0eff36fa48484403fa"
Oct 30 13:12:40.615617 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:12:40.615596    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3948c1f7026fbd363f44018737d9446573ad73eedd30b0eff36fa48484403fa\": container with ID starting with f3948c1f7026fbd363f44018737d9446573ad73eedd30b0eff36fa48484403fa not found: ID does not exist" containerID="f3948c1f7026fbd363f44018737d9446573ad73eedd30b0eff36fa48484403fa"
Oct 30 13:12:40.615660 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:40.615626    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3948c1f7026fbd363f44018737d9446573ad73eedd30b0eff36fa48484403fa"} err="failed to get container status \"f3948c1f7026fbd363f44018737d9446573ad73eedd30b0eff36fa48484403fa\": rpc error: code = NotFound desc = could not find container \"f3948c1f7026fbd363f44018737d9446573ad73eedd30b0eff36fa48484403fa\": container with ID starting with f3948c1f7026fbd363f44018737d9446573ad73eedd30b0eff36fa48484403fa not found: ID does not exist"
Oct 30 13:12:40.660037 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:40.659954    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-662ed421-84ddd86fb7-lz7dl"]
Oct 30 13:12:40.674346 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:40.674308    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-662ed421-84ddd86fb7-lz7dl"]
Oct 30 13:12:41.005752 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:41.005673    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6babd5ad-925c-4ac6-ab8a-9dffbb090259" path="/var/lib/kubelet/pods/6babd5ad-925c-4ac6-ab8a-9dffbb090259/volumes"
Oct 30 13:12:41.006124 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:41.006110    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a54fa560-1acc-4d98-adea-3648e998da77" path="/var/lib/kubelet/pods/a54fa560-1acc-4d98-adea-3648e998da77/volumes"
Oct 30 13:12:41.034806 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:41.034773    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-5852ff-7899c595c5-2z2l5"]
Oct 30 13:12:41.035264 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:41.035250    5093 cpu_manager.go:414] "containerMap: removing container" podUID="6babd5ad-925c-4ac6-ab8a-9dffbb090259" containerName="istio-proxy"
Oct 30 13:12:41.035318 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:41.035265    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6babd5ad-925c-4ac6-ab8a-9dffbb090259" containerName="istio-proxy"
Oct 30 13:12:41.035318 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:41.035300    5093 cpu_manager.go:414] "containerMap: removing container" podUID="6babd5ad-925c-4ac6-ab8a-9dffbb090259" containerName="istio-proxy"
Oct 30 13:12:41.035318 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:41.035306    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6babd5ad-925c-4ac6-ab8a-9dffbb090259" containerName="istio-proxy"
Oct 30 13:12:41.035318 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:41.035313    5093 cpu_manager.go:414] "containerMap: removing container" podUID="f4d7901b-2969-4fb4-9805-f87f7f2a82b1" containerName="payload-pre-processing"
Oct 30 13:12:41.035318 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:41.035318    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4d7901b-2969-4fb4-9805-f87f7f2a82b1" containerName="payload-pre-processing"
Oct 30 13:12:41.035462 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:41.035327    5093 cpu_manager.go:414] "containerMap: removing container" podUID="a54fa560-1acc-4d98-adea-3648e998da77" containerName="maas-api"
Oct 30 13:12:41.035462 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:41.035332    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a54fa560-1acc-4d98-adea-3648e998da77" containerName="maas-api"
Oct 30 13:12:41.035462 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:41.035340    5093 cpu_manager.go:414] "containerMap: removing container" podUID="3e9eef72-38bd-4fea-bd12-36d611a3c933" containerName="payload-pre-processing"
Oct 30 13:12:41.035462 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:41.035345    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e9eef72-38bd-4fea-bd12-36d611a3c933" containerName="payload-pre-processing"
Oct 30 13:12:41.035462 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:41.035415    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="6babd5ad-925c-4ac6-ab8a-9dffbb090259" containerName="istio-proxy"
Oct 30 13:12:41.035462 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:41.035422    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4d7901b-2969-4fb4-9805-f87f7f2a82b1" containerName="payload-pre-processing"
Oct 30 13:12:41.035462 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:41.035429    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="a54fa560-1acc-4d98-adea-3648e998da77" containerName="maas-api"
Oct 30 13:12:41.035462 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:41.035435    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e9eef72-38bd-4fea-bd12-36d611a3c933" containerName="payload-pre-processing"
Oct 30 13:12:41.040160 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:41.040120    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5852ff-7899c595c5-2z2l5"
Oct 30 13:12:41.055882 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:41.055853    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-5852ff\""
Oct 30 13:12:41.055882 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:41.055863    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-5852ff-dockercfg-zxwc5\""
Oct 30 13:12:41.058400 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:41.058370    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-5852ff-7899c595c5-2z2l5"]
Oct 30 13:12:41.127654 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:41.127609    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bndb\" (UniqueName: \"kubernetes.io/projected/db92ac02-6454-4587-a1ec-f98b92646dfb-kube-api-access-8bndb\") pod \"payload-pre-processing-e2e-worker-w3-5852ff-7899c595c5-2z2l5\" (UID: \"db92ac02-6454-4587-a1ec-f98b92646dfb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5852ff-7899c595c5-2z2l5"
Oct 30 13:12:41.128031 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:41.127673    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db92ac02-6454-4587-a1ec-f98b92646dfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-5852ff-7899c595c5-2z2l5\" (UID: \"db92ac02-6454-4587-a1ec-f98b92646dfb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5852ff-7899c595c5-2z2l5"
Oct 30 13:12:41.138981 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:41.138940    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-5852ff-6759c64f6-5vvbb"]
Oct 30 13:12:41.139619 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:41.139598    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="6babd5ad-925c-4ac6-ab8a-9dffbb090259" containerName="istio-proxy"
Oct 30 13:12:41.143466 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:41.143440    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-5852ff-6759c64f6-5vvbb"
Oct 30 13:12:41.158570 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:41.158542    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-5852ff-6759c64f6-5vvbb"]
Oct 30 13:12:41.228779 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:41.228739    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2hjn\" (UniqueName: \"kubernetes.io/projected/3d17739e-fca2-4b2b-aa88-56f7d3a3399e-kube-api-access-c2hjn\") pod \"payload-processing-e2e-worker-w3-5852ff-6759c64f6-5vvbb\" (UID: \"3d17739e-fca2-4b2b-aa88-56f7d3a3399e\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-5852ff-6759c64f6-5vvbb"
Oct 30 13:12:41.229185 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:41.228804    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bndb\" (UniqueName: \"kubernetes.io/projected/db92ac02-6454-4587-a1ec-f98b92646dfb-kube-api-access-8bndb\") pod \"payload-pre-processing-e2e-worker-w3-5852ff-7899c595c5-2z2l5\" (UID: \"db92ac02-6454-4587-a1ec-f98b92646dfb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5852ff-7899c595c5-2z2l5"
Oct 30 13:12:41.229185 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:41.228847    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db92ac02-6454-4587-a1ec-f98b92646dfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-5852ff-7899c595c5-2z2l5\" (UID: \"db92ac02-6454-4587-a1ec-f98b92646dfb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5852ff-7899c595c5-2z2l5"
Oct 30 13:12:41.229185 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:41.228901    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d17739e-fca2-4b2b-aa88-56f7d3a3399e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-5852ff-6759c64f6-5vvbb\" (UID: \"3d17739e-fca2-4b2b-aa88-56f7d3a3399e\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-5852ff-6759c64f6-5vvbb"
Oct 30 13:12:41.229630 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:41.229610    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db92ac02-6454-4587-a1ec-f98b92646dfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-5852ff-7899c595c5-2z2l5\" (UID: \"db92ac02-6454-4587-a1ec-f98b92646dfb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5852ff-7899c595c5-2z2l5"
Oct 30 13:12:41.249619 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:41.249584    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bndb\" (UniqueName: \"kubernetes.io/projected/db92ac02-6454-4587-a1ec-f98b92646dfb-kube-api-access-8bndb\") pod \"payload-pre-processing-e2e-worker-w3-5852ff-7899c595c5-2z2l5\" (UID: \"db92ac02-6454-4587-a1ec-f98b92646dfb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5852ff-7899c595c5-2z2l5"
Oct 30 13:12:41.329828 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:41.329734    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d17739e-fca2-4b2b-aa88-56f7d3a3399e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-5852ff-6759c64f6-5vvbb\" (UID: \"3d17739e-fca2-4b2b-aa88-56f7d3a3399e\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-5852ff-6759c64f6-5vvbb"
Oct 30 13:12:41.330018 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:41.329899    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2hjn\" (UniqueName: \"kubernetes.io/projected/3d17739e-fca2-4b2b-aa88-56f7d3a3399e-kube-api-access-c2hjn\") pod \"payload-processing-e2e-worker-w3-5852ff-6759c64f6-5vvbb\" (UID: \"3d17739e-fca2-4b2b-aa88-56f7d3a3399e\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-5852ff-6759c64f6-5vvbb"
Oct 30 13:12:41.330563 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:41.330536    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d17739e-fca2-4b2b-aa88-56f7d3a3399e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-5852ff-6759c64f6-5vvbb\" (UID: \"3d17739e-fca2-4b2b-aa88-56f7d3a3399e\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-5852ff-6759c64f6-5vvbb"
Oct 30 13:12:41.350561 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:41.350521    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2hjn\" (UniqueName: \"kubernetes.io/projected/3d17739e-fca2-4b2b-aa88-56f7d3a3399e-kube-api-access-c2hjn\") pod \"payload-processing-e2e-worker-w3-5852ff-6759c64f6-5vvbb\" (UID: \"3d17739e-fca2-4b2b-aa88-56f7d3a3399e\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-5852ff-6759c64f6-5vvbb"
Oct 30 13:12:41.354224 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:41.354197    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5852ff-7899c595c5-2z2l5"
Oct 30 13:12:41.459507 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:41.459271    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-5852ff-6759c64f6-5vvbb"
Oct 30 13:12:41.538749 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:41.538699    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-deb5d83f-9bcf86c6c-87gkt" podUID="852636f3-4027-4ba9-a54a-9bb97658752a" containerName="payload-pre-processing" containerID="cri-o://13a93e1af4cf082502d0be81c3665ebf71b9af6b5a93f236eaf94189ac6b0a0e" gracePeriod=25
Oct 30 13:12:41.673701 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:41.673670    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-5852ff-7899c595c5-2z2l5"]
Oct 30 13:12:41.675241 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:12:41.675209    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb92ac02_6454_4587_a1ec_f98b92646dfb.slice/crio-4791f34f3edaadf526afadf52547c4fa923bc0b3565ed311548bb00110e04304 WatchSource:0}: Error finding container 4791f34f3edaadf526afadf52547c4fa923bc0b3565ed311548bb00110e04304: Status 404 returned error can't find the container with id 4791f34f3edaadf526afadf52547c4fa923bc0b3565ed311548bb00110e04304
Oct 30 13:12:41.785004 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:41.784973    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-5852ff-6759c64f6-5vvbb"]
Oct 30 13:12:41.819941 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:41.819915    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-deb5d83f-9bcf86c6c-87gkt"
Oct 30 13:12:41.834489 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:41.834457    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/852636f3-4027-4ba9-a54a-9bb97658752a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/852636f3-4027-4ba9-a54a-9bb97658752a-plugins-config-volume\") pod \"852636f3-4027-4ba9-a54a-9bb97658752a\" (UID: \"852636f3-4027-4ba9-a54a-9bb97658752a\") "
Oct 30 13:12:41.834659 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:41.834525    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/852636f3-4027-4ba9-a54a-9bb97658752a-kube-api-access-dx9bk\" (UniqueName: \"kubernetes.io/projected/852636f3-4027-4ba9-a54a-9bb97658752a-kube-api-access-dx9bk\") pod \"852636f3-4027-4ba9-a54a-9bb97658752a\" (UID: \"852636f3-4027-4ba9-a54a-9bb97658752a\") "
Oct 30 13:12:41.834922 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:41.834890    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/852636f3-4027-4ba9-a54a-9bb97658752a-plugins-config-volume" pod "852636f3-4027-4ba9-a54a-9bb97658752a" (UID: "852636f3-4027-4ba9-a54a-9bb97658752a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:12:41.846820 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:41.846793    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/852636f3-4027-4ba9-a54a-9bb97658752a-kube-api-access-dx9bk" pod "852636f3-4027-4ba9-a54a-9bb97658752a" (UID: "852636f3-4027-4ba9-a54a-9bb97658752a"). InnerVolumeSpecName "kube-api-access-dx9bk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:12:41.935760 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:41.935723    5093 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/852636f3-4027-4ba9-a54a-9bb97658752a-plugins-config-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:12:41.935760 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:41.935755    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx9bk\" (UniqueName: \"kubernetes.io/projected/852636f3-4027-4ba9-a54a-9bb97658752a-kube-api-access-dx9bk\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:12:42.604155 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:42.604034    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-fb3b54cb-7dbc6cbff8-s2dgh"]
Oct 30 13:12:42.624177 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:42.624112    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-5852ff-6759c64f6-5vvbb" event={"ID":"3d17739e-fca2-4b2b-aa88-56f7d3a3399e","Type":"ContainerStarted","Data":"9f8a03054677d3a0401e3f89aedfdb00bfa9ddad18e4fdf5a77875c8d2284d64"}
Oct 30 13:12:42.624362 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:42.624207    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-5852ff-6759c64f6-5vvbb" event={"ID":"3d17739e-fca2-4b2b-aa88-56f7d3a3399e","Type":"ContainerStarted","Data":"1c283ceff9e109949f0fc2c9d7be795b7b3cee1393ed08561afad42681a0d08c"}
Oct 30 13:12:42.624362 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:42.624235    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-5852ff-6759c64f6-5vvbb"
Oct 30 13:12:42.625971 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:42.625936    5093 generic.go:356] "Generic (PLEG): container finished" podID="852636f3-4027-4ba9-a54a-9bb97658752a" containerID="13a93e1af4cf082502d0be81c3665ebf71b9af6b5a93f236eaf94189ac6b0a0e" exitCode=0
Oct 30 13:12:42.626110 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:42.625978    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-deb5d83f-9bcf86c6c-87gkt"
Oct 30 13:12:42.626110 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:42.626012    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-deb5d83f-9bcf86c6c-87gkt" event={"ID":"852636f3-4027-4ba9-a54a-9bb97658752a","Type":"ContainerDied","Data":"13a93e1af4cf082502d0be81c3665ebf71b9af6b5a93f236eaf94189ac6b0a0e"}
Oct 30 13:12:42.626110 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:42.626066    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-deb5d83f-9bcf86c6c-87gkt" event={"ID":"852636f3-4027-4ba9-a54a-9bb97658752a","Type":"ContainerDied","Data":"647160cdfa14da20a69376655a0899f7687ed00c611f6ae854b79b544a294e90"}
Oct 30 13:12:42.626110 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:42.626091    5093 scope.go:122] "RemoveContainer" containerID="13a93e1af4cf082502d0be81c3665ebf71b9af6b5a93f236eaf94189ac6b0a0e"
Oct 30 13:12:42.628279 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:42.628250    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5852ff-7899c595c5-2z2l5" event={"ID":"db92ac02-6454-4587-a1ec-f98b92646dfb","Type":"ContainerStarted","Data":"dad4d976b7c5af384d1a04fe02f4c2770a02b9c7d3f2b4f66a2c623ed240d7c3"}
Oct 30 13:12:42.628427 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:42.628304    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5852ff-7899c595c5-2z2l5" event={"ID":"db92ac02-6454-4587-a1ec-f98b92646dfb","Type":"ContainerStarted","Data":"4791f34f3edaadf526afadf52547c4fa923bc0b3565ed311548bb00110e04304"}
Oct 30 13:12:42.628427 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:42.628407    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5852ff-7899c595c5-2z2l5"
Oct 30 13:12:42.629054 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:42.629036    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fb3b54cb-8676cfd875-p8rhj"]
Oct 30 13:12:42.636722 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:42.636692    5093 scope.go:122] "RemoveContainer" containerID="13a93e1af4cf082502d0be81c3665ebf71b9af6b5a93f236eaf94189ac6b0a0e"
Oct 30 13:12:42.637029 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:12:42.637004    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13a93e1af4cf082502d0be81c3665ebf71b9af6b5a93f236eaf94189ac6b0a0e\": container with ID starting with 13a93e1af4cf082502d0be81c3665ebf71b9af6b5a93f236eaf94189ac6b0a0e not found: ID does not exist" containerID="13a93e1af4cf082502d0be81c3665ebf71b9af6b5a93f236eaf94189ac6b0a0e"
Oct 30 13:12:42.637110 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:42.637041    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13a93e1af4cf082502d0be81c3665ebf71b9af6b5a93f236eaf94189ac6b0a0e"} err="failed to get container status \"13a93e1af4cf082502d0be81c3665ebf71b9af6b5a93f236eaf94189ac6b0a0e\": rpc error: code = NotFound desc = could not find container \"13a93e1af4cf082502d0be81c3665ebf71b9af6b5a93f236eaf94189ac6b0a0e\": container with ID starting with 13a93e1af4cf082502d0be81c3665ebf71b9af6b5a93f236eaf94189ac6b0a0e not found: ID does not exist"
Oct 30 13:12:42.689253 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:42.687877    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-deb5d83f-9bcf86c6c-87gkt"]
Oct 30 13:12:42.699740 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:42.699703    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-deb5d83f-9bcf86c6c-87gkt"]
Oct 30 13:12:42.737064 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:42.737003    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-5852ff-6759c64f6-5vvbb" podStartSLOduration=1.736987873 podStartE2EDuration="1.736987873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:42.735809026 +0000 UTC m=+581.883076267" watchObservedRunningTime="2026-10-30 13:12:42.736987873 +0000 UTC m=+581.884255113"
Oct 30 13:12:42.779107 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:42.779044    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5852ff-7899c595c5-2z2l5" podStartSLOduration=1.779029466 podStartE2EDuration="1.779029466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:42.776718948 +0000 UTC m=+581.923986189" watchObservedRunningTime="2026-10-30 13:12:42.779029466 +0000 UTC m=+581.926296706"
Oct 30 13:12:42.999948 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:42.999915    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="852636f3-4027-4ba9-a54a-9bb97658752a" path="/var/lib/kubelet/pods/852636f3-4027-4ba9-a54a-9bb97658752a/volumes"
Oct 30 13:12:43.780024 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:43.779984    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-ec41e5-7b9c4d99f-v87cn"]
Oct 30 13:12:43.780680 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:43.780655    5093 cpu_manager.go:414] "containerMap: removing container" podUID="852636f3-4027-4ba9-a54a-9bb97658752a" containerName="payload-pre-processing"
Oct 30 13:12:43.780758 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:43.780682    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="852636f3-4027-4ba9-a54a-9bb97658752a" containerName="payload-pre-processing"
Oct 30 13:12:43.780812 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:43.780794    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="852636f3-4027-4ba9-a54a-9bb97658752a" containerName="payload-pre-processing"
Oct 30 13:12:43.784296 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:43.784279    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ec41e5-7b9c4d99f-v87cn"
Oct 30 13:12:43.800173 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:43.800140    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-ec41e5-7b9c4d99f-v87cn"]
Oct 30 13:12:43.809815 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:43.809785    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-ec41e5\""
Oct 30 13:12:43.856202 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:43.856159    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6gpp\" (UniqueName: \"kubernetes.io/projected/7c86ceb7-5877-4c9a-988a-69e8217e6e08-kube-api-access-w6gpp\") pod \"payload-pre-processing-e2e-worker-w2-ec41e5-7b9c4d99f-v87cn\" (UID: \"7c86ceb7-5877-4c9a-988a-69e8217e6e08\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ec41e5-7b9c4d99f-v87cn"
Oct 30 13:12:43.856376 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:43.856238    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c86ceb7-5877-4c9a-988a-69e8217e6e08-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-ec41e5-7b9c4d99f-v87cn\" (UID: \"7c86ceb7-5877-4c9a-988a-69e8217e6e08\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ec41e5-7b9c4d99f-v87cn"
Oct 30 13:12:43.956810 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:43.956773    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6gpp\" (UniqueName: \"kubernetes.io/projected/7c86ceb7-5877-4c9a-988a-69e8217e6e08-kube-api-access-w6gpp\") pod \"payload-pre-processing-e2e-worker-w2-ec41e5-7b9c4d99f-v87cn\" (UID: \"7c86ceb7-5877-4c9a-988a-69e8217e6e08\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ec41e5-7b9c4d99f-v87cn"
Oct 30 13:12:43.956964 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:43.956814    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c86ceb7-5877-4c9a-988a-69e8217e6e08-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-ec41e5-7b9c4d99f-v87cn\" (UID: \"7c86ceb7-5877-4c9a-988a-69e8217e6e08\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ec41e5-7b9c4d99f-v87cn"
Oct 30 13:12:43.957525 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:43.957506    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c86ceb7-5877-4c9a-988a-69e8217e6e08-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-ec41e5-7b9c4d99f-v87cn\" (UID: \"7c86ceb7-5877-4c9a-988a-69e8217e6e08\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ec41e5-7b9c4d99f-v87cn"
Oct 30 13:12:43.978054 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:43.978016    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6gpp\" (UniqueName: \"kubernetes.io/projected/7c86ceb7-5877-4c9a-988a-69e8217e6e08-kube-api-access-w6gpp\") pod \"payload-pre-processing-e2e-worker-w2-ec41e5-7b9c4d99f-v87cn\" (UID: \"7c86ceb7-5877-4c9a-988a-69e8217e6e08\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ec41e5-7b9c4d99f-v87cn"
Oct 30 13:12:44.099604 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.099514    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ec41e5-7b9c4d99f-v87cn"
Oct 30 13:12:44.158252 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.157605    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-5852ff-6dcd476466-j2lxz"]
Oct 30 13:12:44.163477 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.163455    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5852ff-6dcd476466-j2lxz"
Oct 30 13:12:44.187510 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.180597    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:12:44.187510 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.181252    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-5852ff-serving-cert\""
Oct 30 13:12:44.187510 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.181312    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:12:44.187510 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.181327    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 13:12:44.187510 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.181405    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gzx7f\""
Oct 30 13:12:44.187510 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.182201    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-5852ff-6dcd476466-j2lxz"]
Oct 30 13:12:44.261683 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.261646    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a50e3626-0c42-47fc-83c3-a96c1889335c-metrics-certs\") pod \"maas-api-e2e-worker-w3-5852ff-6dcd476466-j2lxz\" (UID: \"a50e3626-0c42-47fc-83c3-a96c1889335c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5852ff-6dcd476466-j2lxz"
Oct 30 13:12:44.261857 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.261722    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2bq8\" (UniqueName: \"kubernetes.io/projected/a50e3626-0c42-47fc-83c3-a96c1889335c-kube-api-access-v2bq8\") pod \"maas-api-e2e-worker-w3-5852ff-6dcd476466-j2lxz\" (UID: \"a50e3626-0c42-47fc-83c3-a96c1889335c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5852ff-6dcd476466-j2lxz"
Oct 30 13:12:44.262014 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.261847    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a50e3626-0c42-47fc-83c3-a96c1889335c-maas-api-tls\") pod \"maas-api-e2e-worker-w3-5852ff-6dcd476466-j2lxz\" (UID: \"a50e3626-0c42-47fc-83c3-a96c1889335c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5852ff-6dcd476466-j2lxz"
Oct 30 13:12:44.262014 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.261911    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a50e3626-0c42-47fc-83c3-a96c1889335c-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-5852ff-6dcd476466-j2lxz\" (UID: \"a50e3626-0c42-47fc-83c3-a96c1889335c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5852ff-6dcd476466-j2lxz"
Oct 30 13:12:44.279358 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.279319    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-662ed421-7dbbd45746-c5rj6" podUID="2cc2a92b-d546-4cca-a0a0-fd23d881a30b" containerName="payload-processing" containerID="cri-o://e3d8f319077a8fbfe46e51b0ef8eff466d354db0d01e5bdac5180643ebbde6c1" gracePeriod=25
Oct 30 13:12:44.307726 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.307661    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-662ed421-5b6df6999f-b69dp" podUID="22ab0145-a9a0-41af-b06a-466519bccd0e" containerName="payload-pre-processing" containerID="cri-o://efcb318f841fc6db5cc49755822d3d407140c41599dcc3701fde49f5d71d9a84" gracePeriod=25
Oct 30 13:12:44.362611 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.362575    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a50e3626-0c42-47fc-83c3-a96c1889335c-metrics-certs\") pod \"maas-api-e2e-worker-w3-5852ff-6dcd476466-j2lxz\" (UID: \"a50e3626-0c42-47fc-83c3-a96c1889335c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5852ff-6dcd476466-j2lxz"
Oct 30 13:12:44.362751 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.362673    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2bq8\" (UniqueName: \"kubernetes.io/projected/a50e3626-0c42-47fc-83c3-a96c1889335c-kube-api-access-v2bq8\") pod \"maas-api-e2e-worker-w3-5852ff-6dcd476466-j2lxz\" (UID: \"a50e3626-0c42-47fc-83c3-a96c1889335c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5852ff-6dcd476466-j2lxz"
Oct 30 13:12:44.362814 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.362761    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a50e3626-0c42-47fc-83c3-a96c1889335c-maas-api-tls\") pod \"maas-api-e2e-worker-w3-5852ff-6dcd476466-j2lxz\" (UID: \"a50e3626-0c42-47fc-83c3-a96c1889335c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5852ff-6dcd476466-j2lxz"
Oct 30 13:12:44.362814 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.362792    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a50e3626-0c42-47fc-83c3-a96c1889335c-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-5852ff-6dcd476466-j2lxz\" (UID: \"a50e3626-0c42-47fc-83c3-a96c1889335c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5852ff-6dcd476466-j2lxz"
Oct 30 13:12:44.363240 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:12:44.363199    5093 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-5852ff-serving-cert: secret "maas-api-e2e-worker-w3-5852ff-serving-cert" not found
Oct 30 13:12:44.363397 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:12:44.363285    5093 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a50e3626-0c42-47fc-83c3-a96c1889335c-maas-api-tls podName:a50e3626-0c42-47fc-83c3-a96c1889335c nodeName:}" failed. No retries permitted until 2026-10-30 13:12:44.863263725 +0000 UTC m=+584.010530946 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a50e3626-0c42-47fc-83c3-a96c1889335c-maas-api-tls") pod "maas-api-e2e-worker-w3-5852ff-6dcd476466-j2lxz" (UID: "a50e3626-0c42-47fc-83c3-a96c1889335c") : secret "maas-api-e2e-worker-w3-5852ff-serving-cert" not found
Oct 30 13:12:44.363642 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.363619    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a50e3626-0c42-47fc-83c3-a96c1889335c-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-5852ff-6dcd476466-j2lxz\" (UID: \"a50e3626-0c42-47fc-83c3-a96c1889335c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5852ff-6dcd476466-j2lxz"
Oct 30 13:12:44.366437 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.366414    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a50e3626-0c42-47fc-83c3-a96c1889335c-metrics-certs\") pod \"maas-api-e2e-worker-w3-5852ff-6dcd476466-j2lxz\" (UID: \"a50e3626-0c42-47fc-83c3-a96c1889335c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5852ff-6dcd476466-j2lxz"
Oct 30 13:12:44.389574 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.389544    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2bq8\" (UniqueName: \"kubernetes.io/projected/a50e3626-0c42-47fc-83c3-a96c1889335c-kube-api-access-v2bq8\") pod \"maas-api-e2e-worker-w3-5852ff-6dcd476466-j2lxz\" (UID: \"a50e3626-0c42-47fc-83c3-a96c1889335c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5852ff-6dcd476466-j2lxz"
Oct 30 13:12:44.412471 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.412437    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-ec41e5-7b9c4d99f-v87cn"]
Oct 30 13:12:44.419287 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:12:44.419257    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c86ceb7_5877_4c9a_988a_69e8217e6e08.slice/crio-889764dd2a62df14eaf9211199661d541279b4fd56a288caddb5517acf176912 WatchSource:0}: Error finding container 889764dd2a62df14eaf9211199661d541279b4fd56a288caddb5517acf176912: Status 404 returned error can't find the container with id 889764dd2a62df14eaf9211199661d541279b4fd56a288caddb5517acf176912
Oct 30 13:12:44.553668 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.553641    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-662ed421-7dbbd45746-c5rj6"
Oct 30 13:12:44.564705 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.564670    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cc2a92b-d546-4cca-a0a0-fd23d881a30b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cc2a92b-d546-4cca-a0a0-fd23d881a30b-plugins-config-volume\") pod \"2cc2a92b-d546-4cca-a0a0-fd23d881a30b\" (UID: \"2cc2a92b-d546-4cca-a0a0-fd23d881a30b\") "
Oct 30 13:12:44.564886 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.564872    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cc2a92b-d546-4cca-a0a0-fd23d881a30b-kube-api-access-kdwcz\" (UniqueName: \"kubernetes.io/projected/2cc2a92b-d546-4cca-a0a0-fd23d881a30b-kube-api-access-kdwcz\") pod \"2cc2a92b-d546-4cca-a0a0-fd23d881a30b\" (UID: \"2cc2a92b-d546-4cca-a0a0-fd23d881a30b\") "
Oct 30 13:12:44.565074 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.565049    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cc2a92b-d546-4cca-a0a0-fd23d881a30b-plugins-config-volume" pod "2cc2a92b-d546-4cca-a0a0-fd23d881a30b" (UID: "2cc2a92b-d546-4cca-a0a0-fd23d881a30b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:12:44.565267 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.565251    5093 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cc2a92b-d546-4cca-a0a0-fd23d881a30b-plugins-config-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:12:44.567657 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.567635    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cc2a92b-d546-4cca-a0a0-fd23d881a30b-kube-api-access-kdwcz" pod "2cc2a92b-d546-4cca-a0a0-fd23d881a30b" (UID: "2cc2a92b-d546-4cca-a0a0-fd23d881a30b"). InnerVolumeSpecName "kube-api-access-kdwcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:12:44.596377 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.596346    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-662ed421-5b6df6999f-b69dp"
Oct 30 13:12:44.646903 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.646812    5093 generic.go:356] "Generic (PLEG): container finished" podID="2cc2a92b-d546-4cca-a0a0-fd23d881a30b" containerID="e3d8f319077a8fbfe46e51b0ef8eff466d354db0d01e5bdac5180643ebbde6c1" exitCode=0
Oct 30 13:12:44.646903 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.646868    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-662ed421-7dbbd45746-c5rj6"
Oct 30 13:12:44.646903 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.646872    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-662ed421-7dbbd45746-c5rj6" event={"ID":"2cc2a92b-d546-4cca-a0a0-fd23d881a30b","Type":"ContainerDied","Data":"e3d8f319077a8fbfe46e51b0ef8eff466d354db0d01e5bdac5180643ebbde6c1"}
Oct 30 13:12:44.647201 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.646945    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-662ed421-7dbbd45746-c5rj6" event={"ID":"2cc2a92b-d546-4cca-a0a0-fd23d881a30b","Type":"ContainerDied","Data":"2fba4d8c4377a4b7c27fc8912a79b35fb9be4a19e6ea0fe243fd0473094c634c"}
Oct 30 13:12:44.647201 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.646978    5093 scope.go:122] "RemoveContainer" containerID="e3d8f319077a8fbfe46e51b0ef8eff466d354db0d01e5bdac5180643ebbde6c1"
Oct 30 13:12:44.649053 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.648967    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ec41e5-7b9c4d99f-v87cn" event={"ID":"7c86ceb7-5877-4c9a-988a-69e8217e6e08","Type":"ContainerStarted","Data":"6c88d7e092420ecd7dde71bc7b57852f68771ff12b3cbd5946ce3b9241cadfd5"}
Oct 30 13:12:44.649053 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.649032    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ec41e5-7b9c4d99f-v87cn" event={"ID":"7c86ceb7-5877-4c9a-988a-69e8217e6e08","Type":"ContainerStarted","Data":"889764dd2a62df14eaf9211199661d541279b4fd56a288caddb5517acf176912"}
Oct 30 13:12:44.649273 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.649068    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ec41e5-7b9c4d99f-v87cn"
Oct 30 13:12:44.651000 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.650971    5093 generic.go:356] "Generic (PLEG): container finished" podID="22ab0145-a9a0-41af-b06a-466519bccd0e" containerID="efcb318f841fc6db5cc49755822d3d407140c41599dcc3701fde49f5d71d9a84" exitCode=0
Oct 30 13:12:44.651113 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.651059    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-662ed421-5b6df6999f-b69dp" event={"ID":"22ab0145-a9a0-41af-b06a-466519bccd0e","Type":"ContainerDied","Data":"efcb318f841fc6db5cc49755822d3d407140c41599dcc3701fde49f5d71d9a84"}
Oct 30 13:12:44.651217 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.651114    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-662ed421-5b6df6999f-b69dp" event={"ID":"22ab0145-a9a0-41af-b06a-466519bccd0e","Type":"ContainerDied","Data":"6feced7b29154bc7468b1f6e3d30710ad072e3e9e4cb424c49cfe6e8d4c2419a"}
Oct 30 13:12:44.651355 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.651332    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-662ed421-5b6df6999f-b69dp"
Oct 30 13:12:44.657831 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.657811    5093 scope.go:122] "RemoveContainer" containerID="e3d8f319077a8fbfe46e51b0ef8eff466d354db0d01e5bdac5180643ebbde6c1"
Oct 30 13:12:44.658110 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:12:44.658089    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3d8f319077a8fbfe46e51b0ef8eff466d354db0d01e5bdac5180643ebbde6c1\": container with ID starting with e3d8f319077a8fbfe46e51b0ef8eff466d354db0d01e5bdac5180643ebbde6c1 not found: ID does not exist" containerID="e3d8f319077a8fbfe46e51b0ef8eff466d354db0d01e5bdac5180643ebbde6c1"
Oct 30 13:12:44.658263 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.658124    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3d8f319077a8fbfe46e51b0ef8eff466d354db0d01e5bdac5180643ebbde6c1"} err="failed to get container status \"e3d8f319077a8fbfe46e51b0ef8eff466d354db0d01e5bdac5180643ebbde6c1\": rpc error: code = NotFound desc = could not find container \"e3d8f319077a8fbfe46e51b0ef8eff466d354db0d01e5bdac5180643ebbde6c1\": container with ID starting with e3d8f319077a8fbfe46e51b0ef8eff466d354db0d01e5bdac5180643ebbde6c1 not found: ID does not exist"
Oct 30 13:12:44.658331 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.658269    5093 scope.go:122] "RemoveContainer" containerID="efcb318f841fc6db5cc49755822d3d407140c41599dcc3701fde49f5d71d9a84"
Oct 30 13:12:44.666634 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.666604    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22ab0145-a9a0-41af-b06a-466519bccd0e-kube-api-access-khgpx\" (UniqueName: \"kubernetes.io/projected/22ab0145-a9a0-41af-b06a-466519bccd0e-kube-api-access-khgpx\") pod \"22ab0145-a9a0-41af-b06a-466519bccd0e\" (UID: \"22ab0145-a9a0-41af-b06a-466519bccd0e\") "
Oct 30 13:12:44.666764 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.666738    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22ab0145-a9a0-41af-b06a-466519bccd0e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22ab0145-a9a0-41af-b06a-466519bccd0e-plugins-config-volume\") pod \"22ab0145-a9a0-41af-b06a-466519bccd0e\" (UID: \"22ab0145-a9a0-41af-b06a-466519bccd0e\") "
Oct 30 13:12:44.667163 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.667111    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22ab0145-a9a0-41af-b06a-466519bccd0e-plugins-config-volume" pod "22ab0145-a9a0-41af-b06a-466519bccd0e" (UID: "22ab0145-a9a0-41af-b06a-466519bccd0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:12:44.667271 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.667223    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdwcz\" (UniqueName: \"kubernetes.io/projected/2cc2a92b-d546-4cca-a0a0-fd23d881a30b-kube-api-access-kdwcz\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:12:44.667271 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.667245    5093 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22ab0145-a9a0-41af-b06a-466519bccd0e-plugins-config-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:12:44.669425 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.669404    5093 scope.go:122] "RemoveContainer" containerID="efcb318f841fc6db5cc49755822d3d407140c41599dcc3701fde49f5d71d9a84"
Oct 30 13:12:44.669532 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.669423    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22ab0145-a9a0-41af-b06a-466519bccd0e-kube-api-access-khgpx" pod "22ab0145-a9a0-41af-b06a-466519bccd0e" (UID: "22ab0145-a9a0-41af-b06a-466519bccd0e"). InnerVolumeSpecName "kube-api-access-khgpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:12:44.669722 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:12:44.669703    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efcb318f841fc6db5cc49755822d3d407140c41599dcc3701fde49f5d71d9a84\": container with ID starting with efcb318f841fc6db5cc49755822d3d407140c41599dcc3701fde49f5d71d9a84 not found: ID does not exist" containerID="efcb318f841fc6db5cc49755822d3d407140c41599dcc3701fde49f5d71d9a84"
Oct 30 13:12:44.669792 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.669728    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efcb318f841fc6db5cc49755822d3d407140c41599dcc3701fde49f5d71d9a84"} err="failed to get container status \"efcb318f841fc6db5cc49755822d3d407140c41599dcc3701fde49f5d71d9a84\": rpc error: code = NotFound desc = could not find container \"efcb318f841fc6db5cc49755822d3d407140c41599dcc3701fde49f5d71d9a84\": container with ID starting with efcb318f841fc6db5cc49755822d3d407140c41599dcc3701fde49f5d71d9a84 not found: ID does not exist"
Oct 30 13:12:44.700669 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.700628    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-662ed421-7dbbd45746-c5rj6"]
Oct 30 13:12:44.703972 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.703946    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-662ed421-7dbbd45746-c5rj6"]
Oct 30 13:12:44.741329 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.741275    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ec41e5-7b9c4d99f-v87cn" podStartSLOduration=1.741257659 podStartE2EDuration="1.741257659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:44.74098451 +0000 UTC m=+583.888251751" watchObservedRunningTime="2026-10-30 13:12:44.741257659 +0000 UTC m=+583.888524901"
Oct 30 13:12:44.768583 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.768549    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khgpx\" (UniqueName: \"kubernetes.io/projected/22ab0145-a9a0-41af-b06a-466519bccd0e-kube-api-access-khgpx\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:12:44.869848 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.869811    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a50e3626-0c42-47fc-83c3-a96c1889335c-maas-api-tls\") pod \"maas-api-e2e-worker-w3-5852ff-6dcd476466-j2lxz\" (UID: \"a50e3626-0c42-47fc-83c3-a96c1889335c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5852ff-6dcd476466-j2lxz"
Oct 30 13:12:44.872800 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:44.872770    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a50e3626-0c42-47fc-83c3-a96c1889335c-maas-api-tls\") pod \"maas-api-e2e-worker-w3-5852ff-6dcd476466-j2lxz\" (UID: \"a50e3626-0c42-47fc-83c3-a96c1889335c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5852ff-6dcd476466-j2lxz"
Oct 30 13:12:45.001565 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:45.001523    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cc2a92b-d546-4cca-a0a0-fd23d881a30b" path="/var/lib/kubelet/pods/2cc2a92b-d546-4cca-a0a0-fd23d881a30b/volumes"
Oct 30 13:12:45.003982 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:45.003957    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-662ed421-5b6df6999f-b69dp"]
Oct 30 13:12:45.006463 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:45.006441    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-662ed421-5b6df6999f-b69dp"]
Oct 30 13:12:45.085229 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:45.085188    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5852ff-6dcd476466-j2lxz"
Oct 30 13:12:45.389909 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:12:45.389846    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda50e3626_0c42_47fc_83c3_a96c1889335c.slice/crio-f54e3df7c0ae5e6e3aa78ae96795ed4c9d4dea624a3416831ed58a31ac3aedcb WatchSource:0}: Error finding container f54e3df7c0ae5e6e3aa78ae96795ed4c9d4dea624a3416831ed58a31ac3aedcb: Status 404 returned error can't find the container with id f54e3df7c0ae5e6e3aa78ae96795ed4c9d4dea624a3416831ed58a31ac3aedcb
Oct 30 13:12:45.390144 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:45.390107    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-5852ff-6dcd476466-j2lxz"]
Oct 30 13:12:45.661853 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:45.661767    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5852ff-6dcd476466-j2lxz" event={"ID":"a50e3626-0c42-47fc-83c3-a96c1889335c","Type":"ContainerStarted","Data":"f54e3df7c0ae5e6e3aa78ae96795ed4c9d4dea624a3416831ed58a31ac3aedcb"}
Oct 30 13:12:46.461755 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:46.461721    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-ec41e5-565987664-cg77b"]
Oct 30 13:12:46.462284 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:46.462267    5093 cpu_manager.go:414] "containerMap: removing container" podUID="2cc2a92b-d546-4cca-a0a0-fd23d881a30b" containerName="payload-processing"
Oct 30 13:12:46.462284 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:46.462285    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cc2a92b-d546-4cca-a0a0-fd23d881a30b" containerName="payload-processing"
Oct 30 13:12:46.462377 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:46.462306    5093 cpu_manager.go:414] "containerMap: removing container" podUID="22ab0145-a9a0-41af-b06a-466519bccd0e" containerName="payload-pre-processing"
Oct 30 13:12:46.462377 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:46.462311    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22ab0145-a9a0-41af-b06a-466519bccd0e" containerName="payload-pre-processing"
Oct 30 13:12:46.462436 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:46.462378    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cc2a92b-d546-4cca-a0a0-fd23d881a30b" containerName="payload-processing"
Oct 30 13:12:46.462436 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:46.462388    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="22ab0145-a9a0-41af-b06a-466519bccd0e" containerName="payload-pre-processing"
Oct 30 13:12:46.465811 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:46.465788    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ec41e5-565987664-cg77b"
Oct 30 13:12:46.490432 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:46.489949    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzx7w\" (UniqueName: \"kubernetes.io/projected/7c30da13-27d5-4947-82e1-21ef1aba9460-kube-api-access-jzx7w\") pod \"maas-api-e2e-worker-w2-ec41e5-565987664-cg77b\" (UID: \"7c30da13-27d5-4947-82e1-21ef1aba9460\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ec41e5-565987664-cg77b"
Oct 30 13:12:46.490432 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:46.490122    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c30da13-27d5-4947-82e1-21ef1aba9460-metrics-certs\") pod \"maas-api-e2e-worker-w2-ec41e5-565987664-cg77b\" (UID: \"7c30da13-27d5-4947-82e1-21ef1aba9460\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ec41e5-565987664-cg77b"
Oct 30 13:12:46.490432 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:46.490206    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c30da13-27d5-4947-82e1-21ef1aba9460-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-ec41e5-565987664-cg77b\" (UID: \"7c30da13-27d5-4947-82e1-21ef1aba9460\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ec41e5-565987664-cg77b"
Oct 30 13:12:46.490432 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:46.490269    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c30da13-27d5-4947-82e1-21ef1aba9460-maas-api-tls\") pod \"maas-api-e2e-worker-w2-ec41e5-565987664-cg77b\" (UID: \"7c30da13-27d5-4947-82e1-21ef1aba9460\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ec41e5-565987664-cg77b"
Oct 30 13:12:46.499071 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:46.499034    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-ec41e5-serving-cert\""
Oct 30 13:12:46.503415 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:46.503379    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-ec41e5-565987664-cg77b"]
Oct 30 13:12:46.591564 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:46.591526    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c30da13-27d5-4947-82e1-21ef1aba9460-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-ec41e5-565987664-cg77b\" (UID: \"7c30da13-27d5-4947-82e1-21ef1aba9460\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ec41e5-565987664-cg77b"
Oct 30 13:12:46.591745 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:46.591591    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c30da13-27d5-4947-82e1-21ef1aba9460-maas-api-tls\") pod \"maas-api-e2e-worker-w2-ec41e5-565987664-cg77b\" (UID: \"7c30da13-27d5-4947-82e1-21ef1aba9460\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ec41e5-565987664-cg77b"
Oct 30 13:12:46.591745 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:46.591638    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzx7w\" (UniqueName: \"kubernetes.io/projected/7c30da13-27d5-4947-82e1-21ef1aba9460-kube-api-access-jzx7w\") pod \"maas-api-e2e-worker-w2-ec41e5-565987664-cg77b\" (UID: \"7c30da13-27d5-4947-82e1-21ef1aba9460\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ec41e5-565987664-cg77b"
Oct 30 13:12:46.591745 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:46.591706    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c30da13-27d5-4947-82e1-21ef1aba9460-metrics-certs\") pod \"maas-api-e2e-worker-w2-ec41e5-565987664-cg77b\" (UID: \"7c30da13-27d5-4947-82e1-21ef1aba9460\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ec41e5-565987664-cg77b"
Oct 30 13:12:46.592353 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:46.592326    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c30da13-27d5-4947-82e1-21ef1aba9460-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-ec41e5-565987664-cg77b\" (UID: \"7c30da13-27d5-4947-82e1-21ef1aba9460\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ec41e5-565987664-cg77b"
Oct 30 13:12:46.594203 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:46.594178    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c30da13-27d5-4947-82e1-21ef1aba9460-maas-api-tls\") pod \"maas-api-e2e-worker-w2-ec41e5-565987664-cg77b\" (UID: \"7c30da13-27d5-4947-82e1-21ef1aba9460\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ec41e5-565987664-cg77b"
Oct 30 13:12:46.594316 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:46.594301    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c30da13-27d5-4947-82e1-21ef1aba9460-metrics-certs\") pod \"maas-api-e2e-worker-w2-ec41e5-565987664-cg77b\" (UID: \"7c30da13-27d5-4947-82e1-21ef1aba9460\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ec41e5-565987664-cg77b"
Oct 30 13:12:46.612103 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:46.612072    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzx7w\" (UniqueName: \"kubernetes.io/projected/7c30da13-27d5-4947-82e1-21ef1aba9460-kube-api-access-jzx7w\") pod \"maas-api-e2e-worker-w2-ec41e5-565987664-cg77b\" (UID: \"7c30da13-27d5-4947-82e1-21ef1aba9460\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ec41e5-565987664-cg77b"
Oct 30 13:12:46.672213 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:46.672170    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5852ff-6dcd476466-j2lxz" event={"ID":"a50e3626-0c42-47fc-83c3-a96c1889335c","Type":"ContainerStarted","Data":"fcb5c5d8a41ec77b4f2a254765da0a5f830d5bf575be354215f96cd781f2d9c9"}
Oct 30 13:12:46.672407 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:46.672314    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5852ff-6dcd476466-j2lxz"
Oct 30 13:12:46.717933 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:46.717802    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5852ff-6dcd476466-j2lxz" podStartSLOduration=2.372758279 podStartE2EDuration="2.717782207s" totalImagesPullingTime="345.023928ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:46.717014949 +0000 UTC m=+585.864282206" watchObservedRunningTime="2026-10-30 13:12:46.717782207 +0000 UTC m=+585.865049445"
Oct 30 13:12:46.780098 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:46.780060    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ec41e5-565987664-cg77b"
Oct 30 13:12:46.999110 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:46.999026    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22ab0145-a9a0-41af-b06a-466519bccd0e" path="/var/lib/kubelet/pods/22ab0145-a9a0-41af-b06a-466519bccd0e/volumes"
Oct 30 13:12:47.093077 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:47.093052    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-ec41e5-565987664-cg77b"]
Oct 30 13:12:47.094697 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:12:47.094654    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c30da13_27d5_4947_82e1_21ef1aba9460.slice/crio-183b74f6c59298cf17d9791b1d559b37ebea13916ad6039ff4ca3e0adf332137 WatchSource:0}: Error finding container 183b74f6c59298cf17d9791b1d559b37ebea13916ad6039ff4ca3e0adf332137: Status 404 returned error can't find the container with id 183b74f6c59298cf17d9791b1d559b37ebea13916ad6039ff4ca3e0adf332137
Oct 30 13:12:47.605010 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:47.604959    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-fb3b54cb-7dbc6cbff8-s2dgh" podUID="550de091-32fb-4959-8f10-cf3c7b43837f" containerName="payload-processing" containerID="cri-o://e7864c3009c65556a3816b7d54a6c5805d0d2fd2c9cc2b87c4fdbb969076c735" gracePeriod=25
Oct 30 13:12:47.629634 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:47.629596    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-fb3b54cb-8676cfd875-p8rhj" podUID="7d13191e-0309-4ff5-87f0-d7366e2c6146" containerName="payload-pre-processing" containerID="cri-o://0d8564ad876329b389db8a79c3d44b361b7dd04ca405b9939d18c22ad48387d6" gracePeriod=25
Oct 30 13:12:47.685280 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:47.685241    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ec41e5-565987664-cg77b" event={"ID":"7c30da13-27d5-4947-82e1-21ef1aba9460","Type":"ContainerStarted","Data":"8f8fc835ed83a43ef9f9518b3d65a29267a8e76dc5e5ba32fed9f4768915f468"}
Oct 30 13:12:47.685455 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:47.685324    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ec41e5-565987664-cg77b"
Oct 30 13:12:47.685455 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:47.685349    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ec41e5-565987664-cg77b" event={"ID":"7c30da13-27d5-4947-82e1-21ef1aba9460","Type":"ContainerStarted","Data":"183b74f6c59298cf17d9791b1d559b37ebea13916ad6039ff4ca3e0adf332137"}
Oct 30 13:12:47.741182 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:47.739311    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ec41e5-565987664-cg77b" podStartSLOduration=1.44324182 podStartE2EDuration="1.739287119s" totalImagesPullingTime="296.045299ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:47.735230847 +0000 UTC m=+586.882498093" watchObservedRunningTime="2026-10-30 13:12:47.739287119 +0000 UTC m=+586.886554361"
Oct 30 13:12:47.982403 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:47.982368    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-fb3b54cb-7dbc6cbff8-s2dgh"
Oct 30 13:12:48.016164 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:48.015052    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/550de091-32fb-4959-8f10-cf3c7b43837f-kube-api-access-mh484\" (UniqueName: \"kubernetes.io/projected/550de091-32fb-4959-8f10-cf3c7b43837f-kube-api-access-mh484\") pod \"550de091-32fb-4959-8f10-cf3c7b43837f\" (UID: \"550de091-32fb-4959-8f10-cf3c7b43837f\") "
Oct 30 13:12:48.016164 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:48.015226    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/550de091-32fb-4959-8f10-cf3c7b43837f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/550de091-32fb-4959-8f10-cf3c7b43837f-plugins-config-volume\") pod \"550de091-32fb-4959-8f10-cf3c7b43837f\" (UID: \"550de091-32fb-4959-8f10-cf3c7b43837f\") "
Oct 30 13:12:48.019963 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:48.019930    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/550de091-32fb-4959-8f10-cf3c7b43837f-plugins-config-volume" pod "550de091-32fb-4959-8f10-cf3c7b43837f" (UID: "550de091-32fb-4959-8f10-cf3c7b43837f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:12:48.038215 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:48.030698    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/550de091-32fb-4959-8f10-cf3c7b43837f-kube-api-access-mh484" pod "550de091-32fb-4959-8f10-cf3c7b43837f" (UID: "550de091-32fb-4959-8f10-cf3c7b43837f"). InnerVolumeSpecName "kube-api-access-mh484". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:12:48.038215 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:48.035869    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-fb3b54cb-8676cfd875-p8rhj"
Oct 30 13:12:48.117011 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:48.116973    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d13191e-0309-4ff5-87f0-d7366e2c6146-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d13191e-0309-4ff5-87f0-d7366e2c6146-plugins-config-volume\") pod \"7d13191e-0309-4ff5-87f0-d7366e2c6146\" (UID: \"7d13191e-0309-4ff5-87f0-d7366e2c6146\") "
Oct 30 13:12:48.117218 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:48.117072    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d13191e-0309-4ff5-87f0-d7366e2c6146-kube-api-access-7ntc5\" (UniqueName: \"kubernetes.io/projected/7d13191e-0309-4ff5-87f0-d7366e2c6146-kube-api-access-7ntc5\") pod \"7d13191e-0309-4ff5-87f0-d7366e2c6146\" (UID: \"7d13191e-0309-4ff5-87f0-d7366e2c6146\") "
Oct 30 13:12:48.117515 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:48.117487    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d13191e-0309-4ff5-87f0-d7366e2c6146-plugins-config-volume" pod "7d13191e-0309-4ff5-87f0-d7366e2c6146" (UID: "7d13191e-0309-4ff5-87f0-d7366e2c6146"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:12:48.117559 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:48.117512    5093 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/550de091-32fb-4959-8f10-cf3c7b43837f-plugins-config-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:12:48.117559 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:48.117537    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh484\" (UniqueName: \"kubernetes.io/projected/550de091-32fb-4959-8f10-cf3c7b43837f-kube-api-access-mh484\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:12:48.119515 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:48.119478    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d13191e-0309-4ff5-87f0-d7366e2c6146-kube-api-access-7ntc5" pod "7d13191e-0309-4ff5-87f0-d7366e2c6146" (UID: "7d13191e-0309-4ff5-87f0-d7366e2c6146"). InnerVolumeSpecName "kube-api-access-7ntc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:12:48.218228 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:48.218124    5093 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d13191e-0309-4ff5-87f0-d7366e2c6146-plugins-config-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:12:48.218228 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:48.218168    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ntc5\" (UniqueName: \"kubernetes.io/projected/7d13191e-0309-4ff5-87f0-d7366e2c6146-kube-api-access-7ntc5\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:12:48.695007 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:48.694967    5093 generic.go:356] "Generic (PLEG): container finished" podID="550de091-32fb-4959-8f10-cf3c7b43837f" containerID="e7864c3009c65556a3816b7d54a6c5805d0d2fd2c9cc2b87c4fdbb969076c735" exitCode=0
Oct 30 13:12:48.695733 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:48.695090    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-fb3b54cb-7dbc6cbff8-s2dgh"
Oct 30 13:12:48.695993 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:48.695099    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-fb3b54cb-7dbc6cbff8-s2dgh" event={"ID":"550de091-32fb-4959-8f10-cf3c7b43837f","Type":"ContainerDied","Data":"e7864c3009c65556a3816b7d54a6c5805d0d2fd2c9cc2b87c4fdbb969076c735"}
Oct 30 13:12:48.696200 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:48.696177    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-fb3b54cb-7dbc6cbff8-s2dgh" event={"ID":"550de091-32fb-4959-8f10-cf3c7b43837f","Type":"ContainerDied","Data":"4f3a8825aeec4d7468573973ea35bf14b71d521e16ce7ad5f32264e32d88393d"}
Oct 30 13:12:48.696372 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:48.696356    5093 scope.go:122] "RemoveContainer" containerID="e7864c3009c65556a3816b7d54a6c5805d0d2fd2c9cc2b87c4fdbb969076c735"
Oct 30 13:12:48.702454 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:48.700089    5093 generic.go:356] "Generic (PLEG): container finished" podID="7d13191e-0309-4ff5-87f0-d7366e2c6146" containerID="0d8564ad876329b389db8a79c3d44b361b7dd04ca405b9939d18c22ad48387d6" exitCode=0
Oct 30 13:12:48.702454 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:48.700917    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-fb3b54cb-8676cfd875-p8rhj"
Oct 30 13:12:48.702454 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:48.702187    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-fb3b54cb-8676cfd875-p8rhj" event={"ID":"7d13191e-0309-4ff5-87f0-d7366e2c6146","Type":"ContainerDied","Data":"0d8564ad876329b389db8a79c3d44b361b7dd04ca405b9939d18c22ad48387d6"}
Oct 30 13:12:48.702454 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:48.702241    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-fb3b54cb-8676cfd875-p8rhj" event={"ID":"7d13191e-0309-4ff5-87f0-d7366e2c6146","Type":"ContainerDied","Data":"14eec9978a35698bf632520ec1909f1a9cf8630940bb09a2ee655afc88b3c4be"}
Oct 30 13:12:48.710824 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:48.710796    5093 scope.go:122] "RemoveContainer" containerID="e7864c3009c65556a3816b7d54a6c5805d0d2fd2c9cc2b87c4fdbb969076c735"
Oct 30 13:12:48.715237 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:12:48.715202    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7864c3009c65556a3816b7d54a6c5805d0d2fd2c9cc2b87c4fdbb969076c735\": container with ID starting with e7864c3009c65556a3816b7d54a6c5805d0d2fd2c9cc2b87c4fdbb969076c735 not found: ID does not exist" containerID="e7864c3009c65556a3816b7d54a6c5805d0d2fd2c9cc2b87c4fdbb969076c735"
Oct 30 13:12:48.715392 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:48.715251    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7864c3009c65556a3816b7d54a6c5805d0d2fd2c9cc2b87c4fdbb969076c735"} err="failed to get container status \"e7864c3009c65556a3816b7d54a6c5805d0d2fd2c9cc2b87c4fdbb969076c735\": rpc error: code = NotFound desc = could not find container \"e7864c3009c65556a3816b7d54a6c5805d0d2fd2c9cc2b87c4fdbb969076c735\": container with ID starting with e7864c3009c65556a3816b7d54a6c5805d0d2fd2c9cc2b87c4fdbb969076c735 not found: ID does not exist"
Oct 30 13:12:48.715392 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:48.715277    5093 scope.go:122] "RemoveContainer" containerID="0d8564ad876329b389db8a79c3d44b361b7dd04ca405b9939d18c22ad48387d6"
Oct 30 13:12:48.727792 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:48.727746    5093 scope.go:122] "RemoveContainer" containerID="0d8564ad876329b389db8a79c3d44b361b7dd04ca405b9939d18c22ad48387d6"
Oct 30 13:12:48.728240 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:12:48.728199    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d8564ad876329b389db8a79c3d44b361b7dd04ca405b9939d18c22ad48387d6\": container with ID starting with 0d8564ad876329b389db8a79c3d44b361b7dd04ca405b9939d18c22ad48387d6 not found: ID does not exist" containerID="0d8564ad876329b389db8a79c3d44b361b7dd04ca405b9939d18c22ad48387d6"
Oct 30 13:12:48.728355 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:48.728243    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d8564ad876329b389db8a79c3d44b361b7dd04ca405b9939d18c22ad48387d6"} err="failed to get container status \"0d8564ad876329b389db8a79c3d44b361b7dd04ca405b9939d18c22ad48387d6\": rpc error: code = NotFound desc = could not find container \"0d8564ad876329b389db8a79c3d44b361b7dd04ca405b9939d18c22ad48387d6\": container with ID starting with 0d8564ad876329b389db8a79c3d44b361b7dd04ca405b9939d18c22ad48387d6 not found: ID does not exist"
Oct 30 13:12:48.759185 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:48.757882    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-fb3b54cb-7dbc6cbff8-s2dgh"]
Oct 30 13:12:48.771204 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:48.771171    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-fb3b54cb-7dbc6cbff8-s2dgh"]
Oct 30 13:12:48.775902 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:12:48.775868    5093 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d13191e_0309_4ff5_87f0_d7366e2c6146.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod550de091_32fb_4959_8f10_cf3c7b43837f.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 13:12:48.810084 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:48.810040    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fb3b54cb-8676cfd875-p8rhj"]
Oct 30 13:12:48.812569 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:48.812543    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fb3b54cb-8676cfd875-p8rhj"]
Oct 30 13:12:48.998536 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:48.998499    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="550de091-32fb-4959-8f10-cf3c7b43837f" path="/var/lib/kubelet/pods/550de091-32fb-4959-8f10-cf3c7b43837f/volumes"
Oct 30 13:12:48.999005 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:48.998985    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d13191e-0309-4ff5-87f0-d7366e2c6146" path="/var/lib/kubelet/pods/7d13191e-0309-4ff5-87f0-d7366e2c6146/volumes"
Oct 30 13:12:49.617973 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:49.617932    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj"]
Oct 30 13:12:49.618731 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:49.618682    5093 cpu_manager.go:414] "containerMap: removing container" podUID="7d13191e-0309-4ff5-87f0-d7366e2c6146" containerName="payload-pre-processing"
Oct 30 13:12:49.618731 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:49.618704    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d13191e-0309-4ff5-87f0-d7366e2c6146" containerName="payload-pre-processing"
Oct 30 13:12:49.618731 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:49.618716    5093 cpu_manager.go:414] "containerMap: removing container" podUID="550de091-32fb-4959-8f10-cf3c7b43837f" containerName="payload-processing"
Oct 30 13:12:49.618731 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:49.618724    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="550de091-32fb-4959-8f10-cf3c7b43837f" containerName="payload-processing"
Oct 30 13:12:49.619055 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:49.618819    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d13191e-0309-4ff5-87f0-d7366e2c6146" containerName="payload-pre-processing"
Oct 30 13:12:49.619055 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:49.618830    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="550de091-32fb-4959-8f10-cf3c7b43837f" containerName="payload-processing"
Oct 30 13:12:49.622785 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:49.622758    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj"
Oct 30 13:12:49.642780 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:49.642750    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj"]
Oct 30 13:12:49.652232 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:49.652201    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-9bf8ad-openshift-default-dockercfg-shhh4\""
Oct 30 13:12:49.734314 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:49.734279    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9667963f-f8d7-4538-b26f-996d05e67f2f-credential-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj\" (UID: \"9667963f-f8d7-4538-b26f-996d05e67f2f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj"
Oct 30 13:12:49.734776 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:49.734354    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9667963f-f8d7-4538-b26f-996d05e67f2f-istio-token\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj\" (UID: \"9667963f-f8d7-4538-b26f-996d05e67f2f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj"
Oct 30 13:12:49.734776 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:49.734408    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9667963f-f8d7-4538-b26f-996d05e67f2f-workload-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj\" (UID: \"9667963f-f8d7-4538-b26f-996d05e67f2f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj"
Oct 30 13:12:49.734776 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:49.734440    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9667963f-f8d7-4538-b26f-996d05e67f2f-istio-podinfo\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj\" (UID: \"9667963f-f8d7-4538-b26f-996d05e67f2f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj"
Oct 30 13:12:49.734776 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:49.734534    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9667963f-f8d7-4538-b26f-996d05e67f2f-istio-envoy\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj\" (UID: \"9667963f-f8d7-4538-b26f-996d05e67f2f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj"
Oct 30 13:12:49.734776 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:49.734556    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9667963f-f8d7-4538-b26f-996d05e67f2f-istiod-ca-cert\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj\" (UID: \"9667963f-f8d7-4538-b26f-996d05e67f2f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj"
Oct 30 13:12:49.734776 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:49.734644    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9667963f-f8d7-4538-b26f-996d05e67f2f-istio-data\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj\" (UID: \"9667963f-f8d7-4538-b26f-996d05e67f2f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj"
Oct 30 13:12:49.734776 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:49.734690    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9667963f-f8d7-4538-b26f-996d05e67f2f-workload-certs\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj\" (UID: \"9667963f-f8d7-4538-b26f-996d05e67f2f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj"
Oct 30 13:12:49.734776 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:49.734748    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqxdz\" (UniqueName: \"kubernetes.io/projected/9667963f-f8d7-4538-b26f-996d05e67f2f-kube-api-access-vqxdz\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj\" (UID: \"9667963f-f8d7-4538-b26f-996d05e67f2f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj"
Oct 30 13:12:49.835757 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:49.835706    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqxdz\" (UniqueName: \"kubernetes.io/projected/9667963f-f8d7-4538-b26f-996d05e67f2f-kube-api-access-vqxdz\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj\" (UID: \"9667963f-f8d7-4538-b26f-996d05e67f2f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj"
Oct 30 13:12:49.835966 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:49.835798    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9667963f-f8d7-4538-b26f-996d05e67f2f-credential-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj\" (UID: \"9667963f-f8d7-4538-b26f-996d05e67f2f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj"
Oct 30 13:12:49.835966 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:49.835853    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9667963f-f8d7-4538-b26f-996d05e67f2f-istio-token\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj\" (UID: \"9667963f-f8d7-4538-b26f-996d05e67f2f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj"
Oct 30 13:12:49.835966 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:49.835886    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9667963f-f8d7-4538-b26f-996d05e67f2f-workload-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj\" (UID: \"9667963f-f8d7-4538-b26f-996d05e67f2f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj"
Oct 30 13:12:49.835966 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:49.835917    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9667963f-f8d7-4538-b26f-996d05e67f2f-istio-podinfo\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj\" (UID: \"9667963f-f8d7-4538-b26f-996d05e67f2f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj"
Oct 30 13:12:49.836207 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:49.836002    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9667963f-f8d7-4538-b26f-996d05e67f2f-istio-envoy\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj\" (UID: \"9667963f-f8d7-4538-b26f-996d05e67f2f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj"
Oct 30 13:12:49.836207 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:49.836033    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9667963f-f8d7-4538-b26f-996d05e67f2f-istiod-ca-cert\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj\" (UID: \"9667963f-f8d7-4538-b26f-996d05e67f2f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj"
Oct 30 13:12:49.836207 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:49.836072    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9667963f-f8d7-4538-b26f-996d05e67f2f-istio-data\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj\" (UID: \"9667963f-f8d7-4538-b26f-996d05e67f2f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj"
Oct 30 13:12:49.836207 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:49.836113    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9667963f-f8d7-4538-b26f-996d05e67f2f-workload-certs\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj\" (UID: \"9667963f-f8d7-4538-b26f-996d05e67f2f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj"
Oct 30 13:12:49.836470 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:49.836445    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9667963f-f8d7-4538-b26f-996d05e67f2f-credential-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj\" (UID: \"9667963f-f8d7-4538-b26f-996d05e67f2f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj"
Oct 30 13:12:49.836470 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:49.836458    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9667963f-f8d7-4538-b26f-996d05e67f2f-workload-certs\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj\" (UID: \"9667963f-f8d7-4538-b26f-996d05e67f2f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj"
Oct 30 13:12:49.836577 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:49.836514    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9667963f-f8d7-4538-b26f-996d05e67f2f-workload-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj\" (UID: \"9667963f-f8d7-4538-b26f-996d05e67f2f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj"
Oct 30 13:12:49.836935 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:49.836909    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9667963f-f8d7-4538-b26f-996d05e67f2f-istio-data\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj\" (UID: \"9667963f-f8d7-4538-b26f-996d05e67f2f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj"
Oct 30 13:12:49.838263 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:49.837833    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9667963f-f8d7-4538-b26f-996d05e67f2f-istiod-ca-cert\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj\" (UID: \"9667963f-f8d7-4538-b26f-996d05e67f2f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj"
Oct 30 13:12:49.839168 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:49.839146    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9667963f-f8d7-4538-b26f-996d05e67f2f-istio-envoy\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj\" (UID: \"9667963f-f8d7-4538-b26f-996d05e67f2f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj"
Oct 30 13:12:49.840094 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:49.840067    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9667963f-f8d7-4538-b26f-996d05e67f2f-istio-podinfo\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj\" (UID: \"9667963f-f8d7-4538-b26f-996d05e67f2f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj"
Oct 30 13:12:49.858867 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:49.858832    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9667963f-f8d7-4538-b26f-996d05e67f2f-istio-token\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj\" (UID: \"9667963f-f8d7-4538-b26f-996d05e67f2f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj"
Oct 30 13:12:49.859140 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:49.859104    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqxdz\" (UniqueName: \"kubernetes.io/projected/9667963f-f8d7-4538-b26f-996d05e67f2f-kube-api-access-vqxdz\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj\" (UID: \"9667963f-f8d7-4538-b26f-996d05e67f2f\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj"
Oct 30 13:12:49.942798 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:49.942749    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj"
Oct 30 13:12:50.261182 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:50.261154    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj"]
Oct 30 13:12:50.262579 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:12:50.262545    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9667963f_f8d7_4538_b26f_996d05e67f2f.slice/crio-65d19f47ac33ad4027d83c9c04fa7f22c5d33b8341e23f393be870a682c075c2 WatchSource:0}: Error finding container 65d19f47ac33ad4027d83c9c04fa7f22c5d33b8341e23f393be870a682c075c2: Status 404 returned error can't find the container with id 65d19f47ac33ad4027d83c9c04fa7f22c5d33b8341e23f393be870a682c075c2
Oct 30 13:12:50.265182 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:50.265099    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:12:50.265274 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:50.265209    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:12:50.265274 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:50.265251    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:12:50.720199 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:50.720164    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj" event={"ID":"9667963f-f8d7-4538-b26f-996d05e67f2f","Type":"ContainerStarted","Data":"cfbdfbb0cf6eccafe1e81e745fccdcac947a8c84e9c1af56c409f5bcb387cec7"}
Oct 30 13:12:50.720373 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:50.720234    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj" event={"ID":"9667963f-f8d7-4538-b26f-996d05e67f2f","Type":"ContainerStarted","Data":"65d19f47ac33ad4027d83c9c04fa7f22c5d33b8341e23f393be870a682c075c2"}
Oct 30 13:12:50.778328 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:50.778268    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj" podStartSLOduration=1.778248552 podStartE2EDuration="1.778248552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:50.775055403 +0000 UTC m=+589.922322646" watchObservedRunningTime="2026-10-30 13:12:50.778248552 +0000 UTC m=+589.925515792"
Oct 30 13:12:50.943524 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:50.943484    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj"
Oct 30 13:12:50.954880 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:50.954843    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c"]
Oct 30 13:12:50.955120 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:50.955094    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c" podUID="7709620c-73e4-4a9e-8b06-ef35865ea56d" containerName="istio-proxy" containerID="cri-o://b8d5a6476076174acb54dccff56db985ee563c5afd1911fb4a445e4e6b7c9930" gracePeriod=30
Oct 30 13:12:51.166931 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:12:51.166843    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:12:51.949230 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:51.949123    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj"
Oct 30 13:12:52.165862 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:52.165815    5093 patch_prober.go:29] interesting pod/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:12:52.166017 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:52.165889    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c" podUID="7709620c-73e4-4a9e-8b06-ef35865ea56d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:12:52.690533 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:52.690499    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5852ff-6dcd476466-j2lxz"
Oct 30 13:12:52.735168 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:52.735116    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj"
Oct 30 13:12:52.736318 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:52.736285    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj"
Oct 30 13:12:53.642216 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:53.642186    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-5852ff-6759c64f6-5vvbb"
Oct 30 13:12:53.642640 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:53.642265    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5852ff-7899c595c5-2z2l5"
Oct 30 13:12:53.708224 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:53.708195    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ec41e5-565987664-cg77b"
Oct 30 13:12:55.666144 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:55.666099    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ec41e5-7b9c4d99f-v87cn"
Oct 30 13:12:56.221620 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:56.221591    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c"
Oct 30 13:12:56.336273 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:56.336174    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7709620c-73e4-4a9e-8b06-ef35865ea56d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7709620c-73e4-4a9e-8b06-ef35865ea56d-istio-data\") pod \"7709620c-73e4-4a9e-8b06-ef35865ea56d\" (UID: \"7709620c-73e4-4a9e-8b06-ef35865ea56d\") "
Oct 30 13:12:56.336273 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:56.336256    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7709620c-73e4-4a9e-8b06-ef35865ea56d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7709620c-73e4-4a9e-8b06-ef35865ea56d-istio-podinfo\") pod \"7709620c-73e4-4a9e-8b06-ef35865ea56d\" (UID: \"7709620c-73e4-4a9e-8b06-ef35865ea56d\") "
Oct 30 13:12:56.336517 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:56.336331    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7709620c-73e4-4a9e-8b06-ef35865ea56d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7709620c-73e4-4a9e-8b06-ef35865ea56d-workload-socket\") pod \"7709620c-73e4-4a9e-8b06-ef35865ea56d\" (UID: \"7709620c-73e4-4a9e-8b06-ef35865ea56d\") "
Oct 30 13:12:56.336517 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:56.336376    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7709620c-73e4-4a9e-8b06-ef35865ea56d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7709620c-73e4-4a9e-8b06-ef35865ea56d-credential-socket\") pod \"7709620c-73e4-4a9e-8b06-ef35865ea56d\" (UID: \"7709620c-73e4-4a9e-8b06-ef35865ea56d\") "
Oct 30 13:12:56.336517 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:56.336408    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7709620c-73e4-4a9e-8b06-ef35865ea56d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7709620c-73e4-4a9e-8b06-ef35865ea56d-istio-envoy\") pod \"7709620c-73e4-4a9e-8b06-ef35865ea56d\" (UID: \"7709620c-73e4-4a9e-8b06-ef35865ea56d\") "
Oct 30 13:12:56.336517 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:56.336444    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7709620c-73e4-4a9e-8b06-ef35865ea56d-istio-data" pod "7709620c-73e4-4a9e-8b06-ef35865ea56d" (UID: "7709620c-73e4-4a9e-8b06-ef35865ea56d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:12:56.336517 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:56.336504    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7709620c-73e4-4a9e-8b06-ef35865ea56d-kube-api-access-rb82v\" (UniqueName: \"kubernetes.io/projected/7709620c-73e4-4a9e-8b06-ef35865ea56d-kube-api-access-rb82v\") pod \"7709620c-73e4-4a9e-8b06-ef35865ea56d\" (UID: \"7709620c-73e4-4a9e-8b06-ef35865ea56d\") "
Oct 30 13:12:56.336777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:56.336575    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7709620c-73e4-4a9e-8b06-ef35865ea56d-istio-token\" (UniqueName: \"kubernetes.io/projected/7709620c-73e4-4a9e-8b06-ef35865ea56d-istio-token\") pod \"7709620c-73e4-4a9e-8b06-ef35865ea56d\" (UID: \"7709620c-73e4-4a9e-8b06-ef35865ea56d\") "
Oct 30 13:12:56.336777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:56.336584    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7709620c-73e4-4a9e-8b06-ef35865ea56d-workload-socket" pod "7709620c-73e4-4a9e-8b06-ef35865ea56d" (UID: "7709620c-73e4-4a9e-8b06-ef35865ea56d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:12:56.336777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:56.336620    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7709620c-73e4-4a9e-8b06-ef35865ea56d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7709620c-73e4-4a9e-8b06-ef35865ea56d-workload-certs\") pod \"7709620c-73e4-4a9e-8b06-ef35865ea56d\" (UID: \"7709620c-73e4-4a9e-8b06-ef35865ea56d\") "
Oct 30 13:12:56.336777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:56.336654    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7709620c-73e4-4a9e-8b06-ef35865ea56d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7709620c-73e4-4a9e-8b06-ef35865ea56d-istiod-ca-cert\") pod \"7709620c-73e4-4a9e-8b06-ef35865ea56d\" (UID: \"7709620c-73e4-4a9e-8b06-ef35865ea56d\") "
Oct 30 13:12:56.336777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:56.336685    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7709620c-73e4-4a9e-8b06-ef35865ea56d-credential-socket" pod "7709620c-73e4-4a9e-8b06-ef35865ea56d" (UID: "7709620c-73e4-4a9e-8b06-ef35865ea56d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:12:56.337031 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:56.336998    5093 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7709620c-73e4-4a9e-8b06-ef35865ea56d-istio-data\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:12:56.337086 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:56.337036    5093 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7709620c-73e4-4a9e-8b06-ef35865ea56d-workload-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:12:56.337086 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:56.337050    5093 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7709620c-73e4-4a9e-8b06-ef35865ea56d-credential-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:12:56.337086 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:56.337058    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7709620c-73e4-4a9e-8b06-ef35865ea56d-workload-certs" pod "7709620c-73e4-4a9e-8b06-ef35865ea56d" (UID: "7709620c-73e4-4a9e-8b06-ef35865ea56d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:12:56.337413 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:56.337389    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7709620c-73e4-4a9e-8b06-ef35865ea56d-istiod-ca-cert" pod "7709620c-73e4-4a9e-8b06-ef35865ea56d" (UID: "7709620c-73e4-4a9e-8b06-ef35865ea56d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:12:56.338684 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:56.338654    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7709620c-73e4-4a9e-8b06-ef35865ea56d-istio-podinfo" pod "7709620c-73e4-4a9e-8b06-ef35865ea56d" (UID: "7709620c-73e4-4a9e-8b06-ef35865ea56d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:12:56.339163 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:56.339114    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7709620c-73e4-4a9e-8b06-ef35865ea56d-istio-token" pod "7709620c-73e4-4a9e-8b06-ef35865ea56d" (UID: "7709620c-73e4-4a9e-8b06-ef35865ea56d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:12:56.339271 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:56.339248    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7709620c-73e4-4a9e-8b06-ef35865ea56d-istio-envoy" pod "7709620c-73e4-4a9e-8b06-ef35865ea56d" (UID: "7709620c-73e4-4a9e-8b06-ef35865ea56d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:12:56.339309 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:56.339250    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7709620c-73e4-4a9e-8b06-ef35865ea56d-kube-api-access-rb82v" pod "7709620c-73e4-4a9e-8b06-ef35865ea56d" (UID: "7709620c-73e4-4a9e-8b06-ef35865ea56d"). InnerVolumeSpecName "kube-api-access-rb82v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:12:56.438269 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:56.438224    5093 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7709620c-73e4-4a9e-8b06-ef35865ea56d-istio-envoy\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:12:56.438269 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:56.438270    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb82v\" (UniqueName: \"kubernetes.io/projected/7709620c-73e4-4a9e-8b06-ef35865ea56d-kube-api-access-rb82v\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:12:56.438552 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:56.438286    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7709620c-73e4-4a9e-8b06-ef35865ea56d-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:12:56.438552 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:56.438299    5093 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7709620c-73e4-4a9e-8b06-ef35865ea56d-workload-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:12:56.438552 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:56.438313    5093 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7709620c-73e4-4a9e-8b06-ef35865ea56d-istiod-ca-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:12:56.438552 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:56.438325    5093 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7709620c-73e4-4a9e-8b06-ef35865ea56d-istio-podinfo\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:12:56.772694 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:56.772659    5093 generic.go:356] "Generic (PLEG): container finished" podID="7709620c-73e4-4a9e-8b06-ef35865ea56d" containerID="b8d5a6476076174acb54dccff56db985ee563c5afd1911fb4a445e4e6b7c9930" exitCode=0
Oct 30 13:12:56.773205 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:56.772733    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c"
Oct 30 13:12:56.773205 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:56.772748    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c" event={"ID":"7709620c-73e4-4a9e-8b06-ef35865ea56d","Type":"ContainerDied","Data":"b8d5a6476076174acb54dccff56db985ee563c5afd1911fb4a445e4e6b7c9930"}
Oct 30 13:12:56.773205 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:56.772859    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c" event={"ID":"7709620c-73e4-4a9e-8b06-ef35865ea56d","Type":"ContainerDied","Data":"486b2b6bec8b5202c3147e355f738f6a59988bf5545d3e04a700301bcaeb104e"}
Oct 30 13:12:56.773205 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:56.772896    5093 scope.go:122] "RemoveContainer" containerID="b8d5a6476076174acb54dccff56db985ee563c5afd1911fb4a445e4e6b7c9930"
Oct 30 13:12:56.784485 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:56.784459    5093 scope.go:122] "RemoveContainer" containerID="7ecc2f9a03aeb4e4c9d7a283e4c56e0a37ec07bd22faf115a220121494399478"
Oct 30 13:12:56.794141 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:56.794109    5093 scope.go:122] "RemoveContainer" containerID="b8d5a6476076174acb54dccff56db985ee563c5afd1911fb4a445e4e6b7c9930"
Oct 30 13:12:56.794437 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:12:56.794409    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8d5a6476076174acb54dccff56db985ee563c5afd1911fb4a445e4e6b7c9930\": container with ID starting with b8d5a6476076174acb54dccff56db985ee563c5afd1911fb4a445e4e6b7c9930 not found: ID does not exist" containerID="b8d5a6476076174acb54dccff56db985ee563c5afd1911fb4a445e4e6b7c9930"
Oct 30 13:12:56.794532 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:56.794443    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8d5a6476076174acb54dccff56db985ee563c5afd1911fb4a445e4e6b7c9930"} err="failed to get container status \"b8d5a6476076174acb54dccff56db985ee563c5afd1911fb4a445e4e6b7c9930\": rpc error: code = NotFound desc = could not find container \"b8d5a6476076174acb54dccff56db985ee563c5afd1911fb4a445e4e6b7c9930\": container with ID starting with b8d5a6476076174acb54dccff56db985ee563c5afd1911fb4a445e4e6b7c9930 not found: ID does not exist"
Oct 30 13:12:56.794532 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:56.794461    5093 scope.go:122] "RemoveContainer" containerID="7ecc2f9a03aeb4e4c9d7a283e4c56e0a37ec07bd22faf115a220121494399478"
Oct 30 13:12:56.794766 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:12:56.794728    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ecc2f9a03aeb4e4c9d7a283e4c56e0a37ec07bd22faf115a220121494399478\": container with ID starting with 7ecc2f9a03aeb4e4c9d7a283e4c56e0a37ec07bd22faf115a220121494399478 not found: ID does not exist" containerID="7ecc2f9a03aeb4e4c9d7a283e4c56e0a37ec07bd22faf115a220121494399478"
Oct 30 13:12:56.794877 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:56.794762    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ecc2f9a03aeb4e4c9d7a283e4c56e0a37ec07bd22faf115a220121494399478"} err="failed to get container status \"7ecc2f9a03aeb4e4c9d7a283e4c56e0a37ec07bd22faf115a220121494399478\": rpc error: code = NotFound desc = could not find container \"7ecc2f9a03aeb4e4c9d7a283e4c56e0a37ec07bd22faf115a220121494399478\": container with ID starting with 7ecc2f9a03aeb4e4c9d7a283e4c56e0a37ec07bd22faf115a220121494399478 not found: ID does not exist"
Oct 30 13:12:56.858322 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:56.858286    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c"]
Oct 30 13:12:56.872556 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:56.872519    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-fb3b54cb-openshift-default-7d4c6fd64f-qgb6c"]
Oct 30 13:12:57.000236 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:12:57.000203    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7709620c-73e4-4a9e-8b06-ef35865ea56d" path="/var/lib/kubelet/pods/7709620c-73e4-4a9e-8b06-ef35865ea56d/volumes"
Oct 30 13:13:01.081661 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:01.081554    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jdj4_f93ee017-ab79-447c-90e0-5b85b71dba24/console-operator/1.log"
Oct 30 13:13:01.083385 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:01.083357    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jdj4_f93ee017-ab79-447c-90e0-5b85b71dba24/console-operator/1.log"
Oct 30 13:13:01.092360 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:01.092210    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tnm_3a675ac7-fbe1-449b-accf-bb0456ed0bd3/ovn-acl-logging/0.log"
Oct 30 13:13:01.093517 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:01.093494    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tnm_3a675ac7-fbe1-449b-accf-bb0456ed0bd3/ovn-acl-logging/0.log"
Oct 30 13:13:01.142287 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:01.142258    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-9bf8ad-6d48f89c8c-wzj8b"]
Oct 30 13:13:01.142775 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:01.142756    5093 cpu_manager.go:414] "containerMap: removing container" podUID="7709620c-73e4-4a9e-8b06-ef35865ea56d" containerName="istio-proxy"
Oct 30 13:13:01.142775 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:01.142771    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7709620c-73e4-4a9e-8b06-ef35865ea56d" containerName="istio-proxy"
Oct 30 13:13:01.142877 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:01.142868    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="7709620c-73e4-4a9e-8b06-ef35865ea56d" containerName="istio-proxy"
Oct 30 13:13:01.142912 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:01.142881    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="7709620c-73e4-4a9e-8b06-ef35865ea56d" containerName="istio-proxy"
Oct 30 13:13:01.152169 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:01.148158    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9bf8ad-6d48f89c8c-wzj8b"
Oct 30 13:13:01.167018 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:01.166992    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-9bf8ad-serving-cert\""
Oct 30 13:13:01.171513 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:01.171485    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-9bf8ad-6d48f89c8c-wzj8b"]
Oct 30 13:13:01.291090 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:01.291055    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75e20d5d-abc6-4682-8b47-0bb3abf6a790-maas-api-tls\") pod \"maas-api-e2e-worker-w4-9bf8ad-6d48f89c8c-wzj8b\" (UID: \"75e20d5d-abc6-4682-8b47-0bb3abf6a790\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9bf8ad-6d48f89c8c-wzj8b"
Oct 30 13:13:01.291300 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:01.291103    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75e20d5d-abc6-4682-8b47-0bb3abf6a790-metrics-certs\") pod \"maas-api-e2e-worker-w4-9bf8ad-6d48f89c8c-wzj8b\" (UID: \"75e20d5d-abc6-4682-8b47-0bb3abf6a790\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9bf8ad-6d48f89c8c-wzj8b"
Oct 30 13:13:01.291300 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:01.291164    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j2mm\" (UniqueName: \"kubernetes.io/projected/75e20d5d-abc6-4682-8b47-0bb3abf6a790-kube-api-access-4j2mm\") pod \"maas-api-e2e-worker-w4-9bf8ad-6d48f89c8c-wzj8b\" (UID: \"75e20d5d-abc6-4682-8b47-0bb3abf6a790\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9bf8ad-6d48f89c8c-wzj8b"
Oct 30 13:13:01.291300 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:01.291274    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75e20d5d-abc6-4682-8b47-0bb3abf6a790-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-9bf8ad-6d48f89c8c-wzj8b\" (UID: \"75e20d5d-abc6-4682-8b47-0bb3abf6a790\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9bf8ad-6d48f89c8c-wzj8b"
Oct 30 13:13:01.392371 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:01.392276    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75e20d5d-abc6-4682-8b47-0bb3abf6a790-maas-api-tls\") pod \"maas-api-e2e-worker-w4-9bf8ad-6d48f89c8c-wzj8b\" (UID: \"75e20d5d-abc6-4682-8b47-0bb3abf6a790\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9bf8ad-6d48f89c8c-wzj8b"
Oct 30 13:13:01.392535 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:01.392401    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75e20d5d-abc6-4682-8b47-0bb3abf6a790-metrics-certs\") pod \"maas-api-e2e-worker-w4-9bf8ad-6d48f89c8c-wzj8b\" (UID: \"75e20d5d-abc6-4682-8b47-0bb3abf6a790\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9bf8ad-6d48f89c8c-wzj8b"
Oct 30 13:13:01.392535 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:01.392459    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j2mm\" (UniqueName: \"kubernetes.io/projected/75e20d5d-abc6-4682-8b47-0bb3abf6a790-kube-api-access-4j2mm\") pod \"maas-api-e2e-worker-w4-9bf8ad-6d48f89c8c-wzj8b\" (UID: \"75e20d5d-abc6-4682-8b47-0bb3abf6a790\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9bf8ad-6d48f89c8c-wzj8b"
Oct 30 13:13:01.392676 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:01.392533    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75e20d5d-abc6-4682-8b47-0bb3abf6a790-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-9bf8ad-6d48f89c8c-wzj8b\" (UID: \"75e20d5d-abc6-4682-8b47-0bb3abf6a790\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9bf8ad-6d48f89c8c-wzj8b"
Oct 30 13:13:01.393437 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:01.393412    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75e20d5d-abc6-4682-8b47-0bb3abf6a790-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-9bf8ad-6d48f89c8c-wzj8b\" (UID: \"75e20d5d-abc6-4682-8b47-0bb3abf6a790\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9bf8ad-6d48f89c8c-wzj8b"
Oct 30 13:13:01.395435 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:01.395397    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75e20d5d-abc6-4682-8b47-0bb3abf6a790-maas-api-tls\") pod \"maas-api-e2e-worker-w4-9bf8ad-6d48f89c8c-wzj8b\" (UID: \"75e20d5d-abc6-4682-8b47-0bb3abf6a790\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9bf8ad-6d48f89c8c-wzj8b"
Oct 30 13:13:01.397619 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:01.397551    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75e20d5d-abc6-4682-8b47-0bb3abf6a790-metrics-certs\") pod \"maas-api-e2e-worker-w4-9bf8ad-6d48f89c8c-wzj8b\" (UID: \"75e20d5d-abc6-4682-8b47-0bb3abf6a790\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9bf8ad-6d48f89c8c-wzj8b"
Oct 30 13:13:01.414635 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:01.414591    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j2mm\" (UniqueName: \"kubernetes.io/projected/75e20d5d-abc6-4682-8b47-0bb3abf6a790-kube-api-access-4j2mm\") pod \"maas-api-e2e-worker-w4-9bf8ad-6d48f89c8c-wzj8b\" (UID: \"75e20d5d-abc6-4682-8b47-0bb3abf6a790\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9bf8ad-6d48f89c8c-wzj8b"
Oct 30 13:13:01.479076 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:01.479039    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9bf8ad-6d48f89c8c-wzj8b"
Oct 30 13:13:01.778515 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:01.778491    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-9bf8ad-6d48f89c8c-wzj8b"]
Oct 30 13:13:01.780377 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:13:01.780342    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75e20d5d_abc6_4682_8b47_0bb3abf6a790.slice/crio-8d2b91d83610aab524e1a862e08daa4032ccc6bb9dc362a5c4950c5af534e154 WatchSource:0}: Error finding container 8d2b91d83610aab524e1a862e08daa4032ccc6bb9dc362a5c4950c5af534e154: Status 404 returned error can't find the container with id 8d2b91d83610aab524e1a862e08daa4032ccc6bb9dc362a5c4950c5af534e154
Oct 30 13:13:01.815592 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:01.815549    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9bf8ad-6d48f89c8c-wzj8b" event={"ID":"75e20d5d-abc6-4682-8b47-0bb3abf6a790","Type":"ContainerStarted","Data":"8d2b91d83610aab524e1a862e08daa4032ccc6bb9dc362a5c4950c5af534e154"}
Oct 30 13:13:02.825894 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:02.825844    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9bf8ad-6d48f89c8c-wzj8b" event={"ID":"75e20d5d-abc6-4682-8b47-0bb3abf6a790","Type":"ContainerStarted","Data":"d92274fdca9a94d2ec274c8f78e377b9d7b83c0ba5b97a70f1251764469cd9dd"}
Oct 30 13:13:02.826375 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:02.825934    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9bf8ad-6d48f89c8c-wzj8b"
Oct 30 13:13:02.868404 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:02.868345    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9bf8ad-6d48f89c8c-wzj8b" podStartSLOduration=1.5542971049999998 podStartE2EDuration="1.868329332s" totalImagesPullingTime="314.032227ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:13:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:13:02.866992191 +0000 UTC m=+602.014259432" watchObservedRunningTime="2026-10-30 13:13:02.868329332 +0000 UTC m=+602.015596574"
Oct 30 13:13:04.849440 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:04.849400    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-52af49-7b74b47477-c9f8d"]
Oct 30 13:13:04.849952 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:04.849925    5093 cpu_manager.go:414] "containerMap: removing container" podUID="7709620c-73e4-4a9e-8b06-ef35865ea56d" containerName="istio-proxy"
Oct 30 13:13:04.849952 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:04.849940    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7709620c-73e4-4a9e-8b06-ef35865ea56d" containerName="istio-proxy"
Oct 30 13:13:04.853657 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:04.853631    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-52af49-7b74b47477-c9f8d"
Oct 30 13:13:04.869969 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:04.869931    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-52af49-serving-cert\""
Oct 30 13:13:04.915107 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:04.908125    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-52af49-7b74b47477-c9f8d"]
Oct 30 13:13:05.032714 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:05.032678    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00cf3b24-1389-4fd2-a8f2-39c6acf5916c-maas-api-tls\") pod \"maas-api-e2e-worker-w1-52af49-7b74b47477-c9f8d\" (UID: \"00cf3b24-1389-4fd2-a8f2-39c6acf5916c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-52af49-7b74b47477-c9f8d"
Oct 30 13:13:05.032891 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:05.032724    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00cf3b24-1389-4fd2-a8f2-39c6acf5916c-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-52af49-7b74b47477-c9f8d\" (UID: \"00cf3b24-1389-4fd2-a8f2-39c6acf5916c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-52af49-7b74b47477-c9f8d"
Oct 30 13:13:05.032891 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:05.032881    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7swk\" (UniqueName: \"kubernetes.io/projected/00cf3b24-1389-4fd2-a8f2-39c6acf5916c-kube-api-access-n7swk\") pod \"maas-api-e2e-worker-w1-52af49-7b74b47477-c9f8d\" (UID: \"00cf3b24-1389-4fd2-a8f2-39c6acf5916c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-52af49-7b74b47477-c9f8d"
Oct 30 13:13:05.033001 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:05.032917    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00cf3b24-1389-4fd2-a8f2-39c6acf5916c-metrics-certs\") pod \"maas-api-e2e-worker-w1-52af49-7b74b47477-c9f8d\" (UID: \"00cf3b24-1389-4fd2-a8f2-39c6acf5916c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-52af49-7b74b47477-c9f8d"
Oct 30 13:13:05.139201 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:05.135722    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7swk\" (UniqueName: \"kubernetes.io/projected/00cf3b24-1389-4fd2-a8f2-39c6acf5916c-kube-api-access-n7swk\") pod \"maas-api-e2e-worker-w1-52af49-7b74b47477-c9f8d\" (UID: \"00cf3b24-1389-4fd2-a8f2-39c6acf5916c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-52af49-7b74b47477-c9f8d"
Oct 30 13:13:05.139201 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:05.135809    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00cf3b24-1389-4fd2-a8f2-39c6acf5916c-metrics-certs\") pod \"maas-api-e2e-worker-w1-52af49-7b74b47477-c9f8d\" (UID: \"00cf3b24-1389-4fd2-a8f2-39c6acf5916c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-52af49-7b74b47477-c9f8d"
Oct 30 13:13:05.139201 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:05.135978    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00cf3b24-1389-4fd2-a8f2-39c6acf5916c-maas-api-tls\") pod \"maas-api-e2e-worker-w1-52af49-7b74b47477-c9f8d\" (UID: \"00cf3b24-1389-4fd2-a8f2-39c6acf5916c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-52af49-7b74b47477-c9f8d"
Oct 30 13:13:05.139201 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:05.136031    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00cf3b24-1389-4fd2-a8f2-39c6acf5916c-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-52af49-7b74b47477-c9f8d\" (UID: \"00cf3b24-1389-4fd2-a8f2-39c6acf5916c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-52af49-7b74b47477-c9f8d"
Oct 30 13:13:05.139201 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:05.139093    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00cf3b24-1389-4fd2-a8f2-39c6acf5916c-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-52af49-7b74b47477-c9f8d\" (UID: \"00cf3b24-1389-4fd2-a8f2-39c6acf5916c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-52af49-7b74b47477-c9f8d"
Oct 30 13:13:05.149933 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:05.149894    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00cf3b24-1389-4fd2-a8f2-39c6acf5916c-maas-api-tls\") pod \"maas-api-e2e-worker-w1-52af49-7b74b47477-c9f8d\" (UID: \"00cf3b24-1389-4fd2-a8f2-39c6acf5916c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-52af49-7b74b47477-c9f8d"
Oct 30 13:13:05.160418 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:05.160382    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00cf3b24-1389-4fd2-a8f2-39c6acf5916c-metrics-certs\") pod \"maas-api-e2e-worker-w1-52af49-7b74b47477-c9f8d\" (UID: \"00cf3b24-1389-4fd2-a8f2-39c6acf5916c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-52af49-7b74b47477-c9f8d"
Oct 30 13:13:05.171345 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:05.167123    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7swk\" (UniqueName: \"kubernetes.io/projected/00cf3b24-1389-4fd2-a8f2-39c6acf5916c-kube-api-access-n7swk\") pod \"maas-api-e2e-worker-w1-52af49-7b74b47477-c9f8d\" (UID: \"00cf3b24-1389-4fd2-a8f2-39c6acf5916c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-52af49-7b74b47477-c9f8d"
Oct 30 13:13:05.171950 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:05.171922    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-52af49-7b74b47477-c9f8d"
Oct 30 13:13:05.496823 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:05.496791    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-52af49-7b74b47477-c9f8d"]
Oct 30 13:13:05.498746 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:13:05.498717    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00cf3b24_1389_4fd2_a8f2_39c6acf5916c.slice/crio-4f5aa88897d9f3a5e3c044d65ebedecbc47906fb4f2c869576bdbb6fa5e4961c WatchSource:0}: Error finding container 4f5aa88897d9f3a5e3c044d65ebedecbc47906fb4f2c869576bdbb6fa5e4961c: Status 404 returned error can't find the container with id 4f5aa88897d9f3a5e3c044d65ebedecbc47906fb4f2c869576bdbb6fa5e4961c
Oct 30 13:13:05.857621 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:05.857408    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-52af49-7b74b47477-c9f8d" event={"ID":"00cf3b24-1389-4fd2-a8f2-39c6acf5916c","Type":"ContainerStarted","Data":"4f5aa88897d9f3a5e3c044d65ebedecbc47906fb4f2c869576bdbb6fa5e4961c"}
Oct 30 13:13:06.867007 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:06.866964    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-52af49-7b74b47477-c9f8d" event={"ID":"00cf3b24-1389-4fd2-a8f2-39c6acf5916c","Type":"ContainerStarted","Data":"f2daf5262312c24efc8e1eed8fdb4181a9b30b15e45b7e7adc79ab6cc345cfca"}
Oct 30 13:13:06.867489 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:06.867053    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-52af49-7b74b47477-c9f8d"
Oct 30 13:13:06.912458 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:06.912372    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-52af49-7b74b47477-c9f8d" podStartSLOduration=2.5552647410000002 podStartE2EDuration="2.912345197s" totalImagesPullingTime="357.080456ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:13:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:13:06.909094577 +0000 UTC m=+606.056361814" watchObservedRunningTime="2026-10-30 13:13:06.912345197 +0000 UTC m=+606.059612441"
Oct 30 13:13:08.862623 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:08.862589    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9bf8ad-6d48f89c8c-wzj8b"
Oct 30 13:13:12.880279 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:12.880242    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-52af49-7b74b47477-c9f8d"
Oct 30 13:13:14.923324 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:14.923283    5093 generic.go:356] "Generic (PLEG): container finished" podID="9667963f-f8d7-4538-b26f-996d05e67f2f" containerID="cfbdfbb0cf6eccafe1e81e745fccdcac947a8c84e9c1af56c409f5bcb387cec7" exitCode=0
Oct 30 13:13:14.923766 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:14.923334    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj" event={"ID":"9667963f-f8d7-4538-b26f-996d05e67f2f","Type":"ContainerDied","Data":"cfbdfbb0cf6eccafe1e81e745fccdcac947a8c84e9c1af56c409f5bcb387cec7"}
Oct 30 13:13:14.923809 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:14.923793    5093 scope.go:122] "RemoveContainer" containerID="cfbdfbb0cf6eccafe1e81e745fccdcac947a8c84e9c1af56c409f5bcb387cec7"
Oct 30 13:13:14.925265 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:14.925231    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:13:14.925374 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:14.925324    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:13:14.925443 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:14.925375    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:13:15.956155 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:15.956030    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj" event={"ID":"9667963f-f8d7-4538-b26f-996d05e67f2f","Type":"ContainerStarted","Data":"d8c17a1e1bf8dc1540c7875542beba21a928737aad8b85ca36607712907117a5"}
Oct 30 13:13:16.946155 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:16.945542    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj"
Oct 30 13:13:16.946155 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:16.945716    5093 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 13:13:16.946155 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:16.945765    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj" podUID="9667963f-f8d7-4538-b26f-996d05e67f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 13:13:17.943597 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:17.943560    5093 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 13:13:17.944232 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:17.944201    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj" podUID="9667963f-f8d7-4538-b26f-996d05e67f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 13:13:18.944040 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:18.944000    5093 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 13:13:18.944702 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:18.944662    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj" podUID="9667963f-f8d7-4538-b26f-996d05e67f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 13:13:19.945600 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:19.945557    5093 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 13:13:19.946245 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:19.946208    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj" podUID="9667963f-f8d7-4538-b26f-996d05e67f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 13:13:20.943919 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:20.943877    5093 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 13:13:20.944142 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:20.943931    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj" podUID="9667963f-f8d7-4538-b26f-996d05e67f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 13:13:21.170461 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:13:21.170401    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:13:21.949055 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:21.949025    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj"
Oct 30 13:13:21.949522 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:21.949497    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj"
Oct 30 13:13:21.950195 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:21.950174    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj"
Oct 30 13:13:26.793386 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:26.793351    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx"]
Oct 30 13:13:26.800229 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:26.800199    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx"
Oct 30 13:13:26.814889 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:26.814860    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx"]
Oct 30 13:13:26.824011 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:26.823980    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9bf8ad\"/\"default-dockercfg-4lf5d\""
Oct 30 13:13:26.824956 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:26.824928    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9bf8ad\"/\"simulator-w4-9bf8ad-kserve-self-signed-certs\""
Oct 30 13:13:26.829811 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:26.829785    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-9bf8ad\"/\"openshift-service-ca.crt\""
Oct 30 13:13:26.829942 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:26.829807    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-9bf8ad\"/\"kube-root-ca.crt\""
Oct 30 13:13:26.897855 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:26.897818    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba156eeb-9955-4f9a-857b-2c4d502a5fcc-home\") pod \"simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx\" (UID: \"ba156eeb-9955-4f9a-857b-2c4d502a5fcc\") " pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx"
Oct 30 13:13:26.898072 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:26.897884    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba156eeb-9955-4f9a-857b-2c4d502a5fcc-tls-certs\") pod \"simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx\" (UID: \"ba156eeb-9955-4f9a-857b-2c4d502a5fcc\") " pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx"
Oct 30 13:13:26.898072 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:26.897983    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba156eeb-9955-4f9a-857b-2c4d502a5fcc-tmp-dir\") pod \"simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx\" (UID: \"ba156eeb-9955-4f9a-857b-2c4d502a5fcc\") " pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx"
Oct 30 13:13:26.898072 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:26.898020    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba156eeb-9955-4f9a-857b-2c4d502a5fcc-model-cache\") pod \"simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx\" (UID: \"ba156eeb-9955-4f9a-857b-2c4d502a5fcc\") " pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx"
Oct 30 13:13:26.898229 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:26.898109    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba156eeb-9955-4f9a-857b-2c4d502a5fcc-dshm\") pod \"simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx\" (UID: \"ba156eeb-9955-4f9a-857b-2c4d502a5fcc\") " pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx"
Oct 30 13:13:26.898229 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:26.898147    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f469c\" (UniqueName: \"kubernetes.io/projected/ba156eeb-9955-4f9a-857b-2c4d502a5fcc-kube-api-access-f469c\") pod \"simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx\" (UID: \"ba156eeb-9955-4f9a-857b-2c4d502a5fcc\") " pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx"
Oct 30 13:13:26.999572 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:26.999539    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba156eeb-9955-4f9a-857b-2c4d502a5fcc-home\") pod \"simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx\" (UID: \"ba156eeb-9955-4f9a-857b-2c4d502a5fcc\") " pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx"
Oct 30 13:13:26.999747 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:26.999603    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba156eeb-9955-4f9a-857b-2c4d502a5fcc-tls-certs\") pod \"simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx\" (UID: \"ba156eeb-9955-4f9a-857b-2c4d502a5fcc\") " pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx"
Oct 30 13:13:26.999747 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:26.999632    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba156eeb-9955-4f9a-857b-2c4d502a5fcc-tmp-dir\") pod \"simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx\" (UID: \"ba156eeb-9955-4f9a-857b-2c4d502a5fcc\") " pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx"
Oct 30 13:13:26.999747 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:26.999653    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba156eeb-9955-4f9a-857b-2c4d502a5fcc-model-cache\") pod \"simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx\" (UID: \"ba156eeb-9955-4f9a-857b-2c4d502a5fcc\") " pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx"
Oct 30 13:13:26.999747 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:26.999720    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba156eeb-9955-4f9a-857b-2c4d502a5fcc-dshm\") pod \"simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx\" (UID: \"ba156eeb-9955-4f9a-857b-2c4d502a5fcc\") " pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx"
Oct 30 13:13:26.999946 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:26.999748    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f469c\" (UniqueName: \"kubernetes.io/projected/ba156eeb-9955-4f9a-857b-2c4d502a5fcc-kube-api-access-f469c\") pod \"simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx\" (UID: \"ba156eeb-9955-4f9a-857b-2c4d502a5fcc\") " pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx"
Oct 30 13:13:27.000006 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:26.999949    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba156eeb-9955-4f9a-857b-2c4d502a5fcc-home\") pod \"simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx\" (UID: \"ba156eeb-9955-4f9a-857b-2c4d502a5fcc\") " pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx"
Oct 30 13:13:27.000062 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:27.000029    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba156eeb-9955-4f9a-857b-2c4d502a5fcc-tmp-dir\") pod \"simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx\" (UID: \"ba156eeb-9955-4f9a-857b-2c4d502a5fcc\") " pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx"
Oct 30 13:13:27.000125 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:27.000065    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba156eeb-9955-4f9a-857b-2c4d502a5fcc-model-cache\") pod \"simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx\" (UID: \"ba156eeb-9955-4f9a-857b-2c4d502a5fcc\") " pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx"
Oct 30 13:13:27.001950 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:27.001919    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba156eeb-9955-4f9a-857b-2c4d502a5fcc-dshm\") pod \"simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx\" (UID: \"ba156eeb-9955-4f9a-857b-2c4d502a5fcc\") " pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx"
Oct 30 13:13:27.002232 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:27.002217    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba156eeb-9955-4f9a-857b-2c4d502a5fcc-tls-certs\") pod \"simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx\" (UID: \"ba156eeb-9955-4f9a-857b-2c4d502a5fcc\") " pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx"
Oct 30 13:13:27.020284 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:27.020250    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f469c\" (UniqueName: \"kubernetes.io/projected/ba156eeb-9955-4f9a-857b-2c4d502a5fcc-kube-api-access-f469c\") pod \"simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx\" (UID: \"ba156eeb-9955-4f9a-857b-2c4d502a5fcc\") " pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx"
Oct 30 13:13:27.119267 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:27.119168    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx"
Oct 30 13:13:27.421406 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:27.421372    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx"]
Oct 30 13:13:27.422808 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:13:27.422777    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba156eeb_9955_4f9a_857b_2c4d502a5fcc.slice/crio-2e50c8bbb0d34c7d5706484c10e3d7885298114e766aabf4975929301e9032c4 WatchSource:0}: Error finding container 2e50c8bbb0d34c7d5706484c10e3d7885298114e766aabf4975929301e9032c4: Status 404 returned error can't find the container with id 2e50c8bbb0d34c7d5706484c10e3d7885298114e766aabf4975929301e9032c4
Oct 30 13:13:28.036763 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:28.036724    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx" event={"ID":"ba156eeb-9955-4f9a-857b-2c4d502a5fcc","Type":"ContainerStarted","Data":"015fe89168d85d6179e16b6f6b6af7cd10529d8d19a277dc8719c3727ff83efa"}
Oct 30 13:13:28.037199 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:28.036776    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx" event={"ID":"ba156eeb-9955-4f9a-857b-2c4d502a5fcc","Type":"ContainerStarted","Data":"2e50c8bbb0d34c7d5706484c10e3d7885298114e766aabf4975929301e9032c4"}
Oct 30 13:13:28.078924 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:28.078870    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx" podStartSLOduration=2.078856696 podStartE2EDuration="2.078856696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:13:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:13:28.077411179 +0000 UTC m=+627.224678421" watchObservedRunningTime="2026-10-30 13:13:28.078856696 +0000 UTC m=+627.226123936"
Oct 30 13:13:28.119622 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:28.119586    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx"
Oct 30 13:13:37.119926 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:37.119886    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx"
Oct 30 13:13:37.136872 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:37.136837    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx"
Oct 30 13:13:38.113648 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:38.113615    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx"
Oct 30 13:13:39.901882 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:39.901835    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt"]
Oct 30 13:13:39.907359 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:39.907331    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt"
Oct 30 13:13:39.924533 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:39.924499    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt"]
Oct 30 13:13:39.934035 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:39.934007    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-5852ff\"/\"kube-root-ca.crt\""
Oct 30 13:13:39.934334 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:39.934303    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5852ff\"/\"premium-w3-5852ff-kserve-self-signed-certs\""
Oct 30 13:13:39.934511 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:39.934489    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-5852ff\"/\"openshift-service-ca.crt\""
Oct 30 13:13:39.936927 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:39.936904    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5852ff\"/\"default-dockercfg-f7c7v\""
Oct 30 13:13:40.051322 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:40.051285    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ae52c69-0c30-438f-9ac0-c87b6de16061-home\") pod \"premium-w3-5852ff-kserve-5c95746977-klqrt\" (UID: \"4ae52c69-0c30-438f-9ac0-c87b6de16061\") " pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt"
Oct 30 13:13:40.051322 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:40.051323    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ae52c69-0c30-438f-9ac0-c87b6de16061-tls-certs\") pod \"premium-w3-5852ff-kserve-5c95746977-klqrt\" (UID: \"4ae52c69-0c30-438f-9ac0-c87b6de16061\") " pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt"
Oct 30 13:13:40.051609 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:40.051361    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ae52c69-0c30-438f-9ac0-c87b6de16061-tmp-dir\") pod \"premium-w3-5852ff-kserve-5c95746977-klqrt\" (UID: \"4ae52c69-0c30-438f-9ac0-c87b6de16061\") " pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt"
Oct 30 13:13:40.051609 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:40.051506    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ltqf\" (UniqueName: \"kubernetes.io/projected/4ae52c69-0c30-438f-9ac0-c87b6de16061-kube-api-access-8ltqf\") pod \"premium-w3-5852ff-kserve-5c95746977-klqrt\" (UID: \"4ae52c69-0c30-438f-9ac0-c87b6de16061\") " pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt"
Oct 30 13:13:40.051737 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:40.051719    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ae52c69-0c30-438f-9ac0-c87b6de16061-dshm\") pod \"premium-w3-5852ff-kserve-5c95746977-klqrt\" (UID: \"4ae52c69-0c30-438f-9ac0-c87b6de16061\") " pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt"
Oct 30 13:13:40.051879 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:40.051856    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ae52c69-0c30-438f-9ac0-c87b6de16061-model-cache\") pod \"premium-w3-5852ff-kserve-5c95746977-klqrt\" (UID: \"4ae52c69-0c30-438f-9ac0-c87b6de16061\") " pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt"
Oct 30 13:13:40.152935 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:40.152842    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ae52c69-0c30-438f-9ac0-c87b6de16061-dshm\") pod \"premium-w3-5852ff-kserve-5c95746977-klqrt\" (UID: \"4ae52c69-0c30-438f-9ac0-c87b6de16061\") " pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt"
Oct 30 13:13:40.153119 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:40.152949    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ae52c69-0c30-438f-9ac0-c87b6de16061-model-cache\") pod \"premium-w3-5852ff-kserve-5c95746977-klqrt\" (UID: \"4ae52c69-0c30-438f-9ac0-c87b6de16061\") " pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt"
Oct 30 13:13:40.153119 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:40.153008    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ae52c69-0c30-438f-9ac0-c87b6de16061-home\") pod \"premium-w3-5852ff-kserve-5c95746977-klqrt\" (UID: \"4ae52c69-0c30-438f-9ac0-c87b6de16061\") " pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt"
Oct 30 13:13:40.153119 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:40.153036    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ae52c69-0c30-438f-9ac0-c87b6de16061-tls-certs\") pod \"premium-w3-5852ff-kserve-5c95746977-klqrt\" (UID: \"4ae52c69-0c30-438f-9ac0-c87b6de16061\") " pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt"
Oct 30 13:13:40.153119 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:40.153078    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ae52c69-0c30-438f-9ac0-c87b6de16061-tmp-dir\") pod \"premium-w3-5852ff-kserve-5c95746977-klqrt\" (UID: \"4ae52c69-0c30-438f-9ac0-c87b6de16061\") " pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt"
Oct 30 13:13:40.153382 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:40.153122    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ltqf\" (UniqueName: \"kubernetes.io/projected/4ae52c69-0c30-438f-9ac0-c87b6de16061-kube-api-access-8ltqf\") pod \"premium-w3-5852ff-kserve-5c95746977-klqrt\" (UID: \"4ae52c69-0c30-438f-9ac0-c87b6de16061\") " pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt"
Oct 30 13:13:40.153451 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:40.153425    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ae52c69-0c30-438f-9ac0-c87b6de16061-home\") pod \"premium-w3-5852ff-kserve-5c95746977-klqrt\" (UID: \"4ae52c69-0c30-438f-9ac0-c87b6de16061\") " pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt"
Oct 30 13:13:40.153578 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:40.153513    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ae52c69-0c30-438f-9ac0-c87b6de16061-tmp-dir\") pod \"premium-w3-5852ff-kserve-5c95746977-klqrt\" (UID: \"4ae52c69-0c30-438f-9ac0-c87b6de16061\") " pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt"
Oct 30 13:13:40.153578 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:40.153531    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ae52c69-0c30-438f-9ac0-c87b6de16061-model-cache\") pod \"premium-w3-5852ff-kserve-5c95746977-klqrt\" (UID: \"4ae52c69-0c30-438f-9ac0-c87b6de16061\") " pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt"
Oct 30 13:13:40.155304 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:40.155282    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ae52c69-0c30-438f-9ac0-c87b6de16061-dshm\") pod \"premium-w3-5852ff-kserve-5c95746977-klqrt\" (UID: \"4ae52c69-0c30-438f-9ac0-c87b6de16061\") " pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt"
Oct 30 13:13:40.155489 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:40.155473    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ae52c69-0c30-438f-9ac0-c87b6de16061-tls-certs\") pod \"premium-w3-5852ff-kserve-5c95746977-klqrt\" (UID: \"4ae52c69-0c30-438f-9ac0-c87b6de16061\") " pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt"
Oct 30 13:13:40.174332 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:40.174298    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ltqf\" (UniqueName: \"kubernetes.io/projected/4ae52c69-0c30-438f-9ac0-c87b6de16061-kube-api-access-8ltqf\") pod \"premium-w3-5852ff-kserve-5c95746977-klqrt\" (UID: \"4ae52c69-0c30-438f-9ac0-c87b6de16061\") " pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt"
Oct 30 13:13:40.224098 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:40.224060    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt"
Oct 30 13:13:40.528167 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:40.528120    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt"]
Oct 30 13:13:40.529199 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:13:40.529164    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ae52c69_0c30_438f_9ac0_c87b6de16061.slice/crio-1a11c3af534ae2a30fc8022fa68ed1702134bbc8f496cf00074a259911225fcc WatchSource:0}: Error finding container 1a11c3af534ae2a30fc8022fa68ed1702134bbc8f496cf00074a259911225fcc: Status 404 returned error can't find the container with id 1a11c3af534ae2a30fc8022fa68ed1702134bbc8f496cf00074a259911225fcc
Oct 30 13:13:41.124617 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:41.124580    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt" event={"ID":"4ae52c69-0c30-438f-9ac0-c87b6de16061","Type":"ContainerStarted","Data":"345309aba8f9a3c65e4b8614e3de3929be5b4858b91f8233b32dae4db5ac2bac"}
Oct 30 13:13:41.124617 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:41.124625    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt" event={"ID":"4ae52c69-0c30-438f-9ac0-c87b6de16061","Type":"ContainerStarted","Data":"1a11c3af534ae2a30fc8022fa68ed1702134bbc8f496cf00074a259911225fcc"}
Oct 30 13:13:41.166363 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:41.166311    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt" podStartSLOduration=2.166296249 podStartE2EDuration="2.166296249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:13:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:13:41.165436287 +0000 UTC m=+640.312703542" watchObservedRunningTime="2026-10-30 13:13:41.166296249 +0000 UTC m=+640.313563489"
Oct 30 13:13:41.225240 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:41.225193    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt"
Oct 30 13:13:50.224951 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:50.224905    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt"
Oct 30 13:13:50.244017 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:50.243978    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt"
Oct 30 13:13:51.167517 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:13:51.167476    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:13:51.205829 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:13:51.205800    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt"
Oct 30 13:14:03.281306 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:14:03.281271    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv"]
Oct 30 13:14:03.287042 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:14:03.287015    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv"
Oct 30 13:14:03.300652 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:14:03.300622    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv"]
Oct 30 13:14:03.309926 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:14:03.309893    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9bf8ad\"/\"premium-w4-9bf8ad-kserve-self-signed-certs\""
Oct 30 13:14:03.413307 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:14:03.413265    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh6h7\" (UniqueName: \"kubernetes.io/projected/d6a44981-104f-4d3e-9937-d184a6ddc482-kube-api-access-lh6h7\") pod \"premium-w4-9bf8ad-kserve-86c698998b-r2jbv\" (UID: \"d6a44981-104f-4d3e-9937-d184a6ddc482\") " pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv"
Oct 30 13:14:03.413489 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:14:03.413381    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6a44981-104f-4d3e-9937-d184a6ddc482-dshm\") pod \"premium-w4-9bf8ad-kserve-86c698998b-r2jbv\" (UID: \"d6a44981-104f-4d3e-9937-d184a6ddc482\") " pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv"
Oct 30 13:14:03.413489 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:14:03.413448    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6a44981-104f-4d3e-9937-d184a6ddc482-tls-certs\") pod \"premium-w4-9bf8ad-kserve-86c698998b-r2jbv\" (UID: \"d6a44981-104f-4d3e-9937-d184a6ddc482\") " pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv"
Oct 30 13:14:03.413597 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:14:03.413491    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6a44981-104f-4d3e-9937-d184a6ddc482-tmp-dir\") pod \"premium-w4-9bf8ad-kserve-86c698998b-r2jbv\" (UID: \"d6a44981-104f-4d3e-9937-d184a6ddc482\") " pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv"
Oct 30 13:14:03.413597 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:14:03.413559    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6a44981-104f-4d3e-9937-d184a6ddc482-home\") pod \"premium-w4-9bf8ad-kserve-86c698998b-r2jbv\" (UID: \"d6a44981-104f-4d3e-9937-d184a6ddc482\") " pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv"
Oct 30 13:14:03.413680 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:14:03.413632    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6a44981-104f-4d3e-9937-d184a6ddc482-model-cache\") pod \"premium-w4-9bf8ad-kserve-86c698998b-r2jbv\" (UID: \"d6a44981-104f-4d3e-9937-d184a6ddc482\") " pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv"
Oct 30 13:14:03.515111 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:14:03.515068    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6a44981-104f-4d3e-9937-d184a6ddc482-tls-certs\") pod \"premium-w4-9bf8ad-kserve-86c698998b-r2jbv\" (UID: \"d6a44981-104f-4d3e-9937-d184a6ddc482\") " pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv"
Oct 30 13:14:03.515364 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:14:03.515152    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6a44981-104f-4d3e-9937-d184a6ddc482-tmp-dir\") pod \"premium-w4-9bf8ad-kserve-86c698998b-r2jbv\" (UID: \"d6a44981-104f-4d3e-9937-d184a6ddc482\") " pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv"
Oct 30 13:14:03.515364 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:14:03.515191    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6a44981-104f-4d3e-9937-d184a6ddc482-home\") pod \"premium-w4-9bf8ad-kserve-86c698998b-r2jbv\" (UID: \"d6a44981-104f-4d3e-9937-d184a6ddc482\") " pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv"
Oct 30 13:14:03.515364 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:14:03.515271    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6a44981-104f-4d3e-9937-d184a6ddc482-model-cache\") pod \"premium-w4-9bf8ad-kserve-86c698998b-r2jbv\" (UID: \"d6a44981-104f-4d3e-9937-d184a6ddc482\") " pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv"
Oct 30 13:14:03.515364 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:14:03.515306    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh6h7\" (UniqueName: \"kubernetes.io/projected/d6a44981-104f-4d3e-9937-d184a6ddc482-kube-api-access-lh6h7\") pod \"premium-w4-9bf8ad-kserve-86c698998b-r2jbv\" (UID: \"d6a44981-104f-4d3e-9937-d184a6ddc482\") " pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv"
Oct 30 13:14:03.515582 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:14:03.515368    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6a44981-104f-4d3e-9937-d184a6ddc482-dshm\") pod \"premium-w4-9bf8ad-kserve-86c698998b-r2jbv\" (UID: \"d6a44981-104f-4d3e-9937-d184a6ddc482\") " pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv"
Oct 30 13:14:03.515965 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:14:03.515652    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6a44981-104f-4d3e-9937-d184a6ddc482-tmp-dir\") pod \"premium-w4-9bf8ad-kserve-86c698998b-r2jbv\" (UID: \"d6a44981-104f-4d3e-9937-d184a6ddc482\") " pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv"
Oct 30 13:14:03.515965 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:14:03.515709    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6a44981-104f-4d3e-9937-d184a6ddc482-home\") pod \"premium-w4-9bf8ad-kserve-86c698998b-r2jbv\" (UID: \"d6a44981-104f-4d3e-9937-d184a6ddc482\") " pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv"
Oct 30 13:14:03.515965 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:14:03.515912    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6a44981-104f-4d3e-9937-d184a6ddc482-model-cache\") pod \"premium-w4-9bf8ad-kserve-86c698998b-r2jbv\" (UID: \"d6a44981-104f-4d3e-9937-d184a6ddc482\") " pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv"
Oct 30 13:14:03.517907 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:14:03.517883    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6a44981-104f-4d3e-9937-d184a6ddc482-tls-certs\") pod \"premium-w4-9bf8ad-kserve-86c698998b-r2jbv\" (UID: \"d6a44981-104f-4d3e-9937-d184a6ddc482\") " pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv"
Oct 30 13:14:03.518125 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:14:03.518107    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6a44981-104f-4d3e-9937-d184a6ddc482-dshm\") pod \"premium-w4-9bf8ad-kserve-86c698998b-r2jbv\" (UID: \"d6a44981-104f-4d3e-9937-d184a6ddc482\") " pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv"
Oct 30 13:14:03.538246 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:14:03.538185    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh6h7\" (UniqueName: \"kubernetes.io/projected/d6a44981-104f-4d3e-9937-d184a6ddc482-kube-api-access-lh6h7\") pod \"premium-w4-9bf8ad-kserve-86c698998b-r2jbv\" (UID: \"d6a44981-104f-4d3e-9937-d184a6ddc482\") " pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv"
Oct 30 13:14:03.601764 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:14:03.601728    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv"
Oct 30 13:14:03.903363 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:14:03.903333    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv"]
Oct 30 13:14:03.905180 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:14:03.905109    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6a44981_104f_4d3e_9937_d184a6ddc482.slice/crio-8878e62d4d3dd14c2621683747e7cd2af1be168f51623b5bf50d1ce073d211bc WatchSource:0}: Error finding container 8878e62d4d3dd14c2621683747e7cd2af1be168f51623b5bf50d1ce073d211bc: Status 404 returned error can't find the container with id 8878e62d4d3dd14c2621683747e7cd2af1be168f51623b5bf50d1ce073d211bc
Oct 30 13:14:04.291521 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:14:04.291486    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv" event={"ID":"d6a44981-104f-4d3e-9937-d184a6ddc482","Type":"ContainerStarted","Data":"916ddc84c1bb68343ff23094fa39c91d7f97250e51f41a27d30895d605577072"}
Oct 30 13:14:04.291924 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:14:04.291533    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv" event={"ID":"d6a44981-104f-4d3e-9937-d184a6ddc482","Type":"ContainerStarted","Data":"8878e62d4d3dd14c2621683747e7cd2af1be168f51623b5bf50d1ce073d211bc"}
Oct 30 13:14:04.335852 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:14:04.335795    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv" podStartSLOduration=1.335781458 podStartE2EDuration="1.335781458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:14:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:14:04.334305397 +0000 UTC m=+663.481572640" watchObservedRunningTime="2026-10-30 13:14:04.335781458 +0000 UTC m=+663.483048700"
Oct 30 13:14:04.602122 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:14:04.602039    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv"
Oct 30 13:14:13.601904 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:14:13.601813    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv"
Oct 30 13:14:13.614495 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:14:13.614469    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv"
Oct 30 13:14:14.385649 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:14:14.385619    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv"
Oct 30 13:14:21.169100 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:14:21.169056    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:14:51.167657 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:14:51.167620    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:15:08.027515 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:08.027478    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk"]
Oct 30 13:15:08.032033 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:08.031988    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk"
Oct 30 13:15:08.083208 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:08.075193    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-ec41e5-openshift-default-dockercfg-6fkcb\""
Oct 30 13:15:08.083208 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:08.075468    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f31be4a0-295f-4ed0-901d-f45cee943496-credential-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk\" (UID: \"f31be4a0-295f-4ed0-901d-f45cee943496\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk"
Oct 30 13:15:08.083208 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:08.075527    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f31be4a0-295f-4ed0-901d-f45cee943496-istio-token\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk\" (UID: \"f31be4a0-295f-4ed0-901d-f45cee943496\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk"
Oct 30 13:15:08.083208 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:08.075665    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f31be4a0-295f-4ed0-901d-f45cee943496-istio-envoy\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk\" (UID: \"f31be4a0-295f-4ed0-901d-f45cee943496\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk"
Oct 30 13:15:08.083208 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:08.075704    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hsms\" (UniqueName: \"kubernetes.io/projected/f31be4a0-295f-4ed0-901d-f45cee943496-kube-api-access-9hsms\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk\" (UID: \"f31be4a0-295f-4ed0-901d-f45cee943496\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk"
Oct 30 13:15:08.083208 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:08.075785    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f31be4a0-295f-4ed0-901d-f45cee943496-workload-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk\" (UID: \"f31be4a0-295f-4ed0-901d-f45cee943496\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk"
Oct 30 13:15:08.083208 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:08.075941    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f31be4a0-295f-4ed0-901d-f45cee943496-workload-certs\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk\" (UID: \"f31be4a0-295f-4ed0-901d-f45cee943496\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk"
Oct 30 13:15:08.083208 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:08.075999    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f31be4a0-295f-4ed0-901d-f45cee943496-istio-data\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk\" (UID: \"f31be4a0-295f-4ed0-901d-f45cee943496\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk"
Oct 30 13:15:08.083208 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:08.076468    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f31be4a0-295f-4ed0-901d-f45cee943496-istio-podinfo\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk\" (UID: \"f31be4a0-295f-4ed0-901d-f45cee943496\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk"
Oct 30 13:15:08.083208 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:08.076507    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f31be4a0-295f-4ed0-901d-f45cee943496-istiod-ca-cert\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk\" (UID: \"f31be4a0-295f-4ed0-901d-f45cee943496\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk"
Oct 30 13:15:08.083208 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:08.082344    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk"]
Oct 30 13:15:08.178144 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:08.178103    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f31be4a0-295f-4ed0-901d-f45cee943496-istio-envoy\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk\" (UID: \"f31be4a0-295f-4ed0-901d-f45cee943496\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk"
Oct 30 13:15:08.178352 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:08.178159    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hsms\" (UniqueName: \"kubernetes.io/projected/f31be4a0-295f-4ed0-901d-f45cee943496-kube-api-access-9hsms\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk\" (UID: \"f31be4a0-295f-4ed0-901d-f45cee943496\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk"
Oct 30 13:15:08.178352 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:08.178195    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f31be4a0-295f-4ed0-901d-f45cee943496-workload-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk\" (UID: \"f31be4a0-295f-4ed0-901d-f45cee943496\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk"
Oct 30 13:15:08.178352 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:08.178307    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f31be4a0-295f-4ed0-901d-f45cee943496-workload-certs\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk\" (UID: \"f31be4a0-295f-4ed0-901d-f45cee943496\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk"
Oct 30 13:15:08.178352 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:08.178346    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f31be4a0-295f-4ed0-901d-f45cee943496-istio-data\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk\" (UID: \"f31be4a0-295f-4ed0-901d-f45cee943496\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk"
Oct 30 13:15:08.178585 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:08.178389    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f31be4a0-295f-4ed0-901d-f45cee943496-istio-podinfo\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk\" (UID: \"f31be4a0-295f-4ed0-901d-f45cee943496\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk"
Oct 30 13:15:08.178655 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:08.178587    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f31be4a0-295f-4ed0-901d-f45cee943496-istiod-ca-cert\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk\" (UID: \"f31be4a0-295f-4ed0-901d-f45cee943496\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk"
Oct 30 13:15:08.178712 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:08.178656    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f31be4a0-295f-4ed0-901d-f45cee943496-workload-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk\" (UID: \"f31be4a0-295f-4ed0-901d-f45cee943496\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk"
Oct 30 13:15:08.178712 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:08.178697    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f31be4a0-295f-4ed0-901d-f45cee943496-credential-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk\" (UID: \"f31be4a0-295f-4ed0-901d-f45cee943496\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk"
Oct 30 13:15:08.178813 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:08.178749    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f31be4a0-295f-4ed0-901d-f45cee943496-istio-token\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk\" (UID: \"f31be4a0-295f-4ed0-901d-f45cee943496\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk"
Oct 30 13:15:08.178813 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:08.178770    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f31be4a0-295f-4ed0-901d-f45cee943496-istio-data\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk\" (UID: \"f31be4a0-295f-4ed0-901d-f45cee943496\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk"
Oct 30 13:15:08.178813 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:08.178703    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f31be4a0-295f-4ed0-901d-f45cee943496-workload-certs\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk\" (UID: \"f31be4a0-295f-4ed0-901d-f45cee943496\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk"
Oct 30 13:15:08.179120 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:08.179025    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f31be4a0-295f-4ed0-901d-f45cee943496-credential-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk\" (UID: \"f31be4a0-295f-4ed0-901d-f45cee943496\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk"
Oct 30 13:15:08.179426 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:08.179396    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f31be4a0-295f-4ed0-901d-f45cee943496-istiod-ca-cert\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk\" (UID: \"f31be4a0-295f-4ed0-901d-f45cee943496\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk"
Oct 30 13:15:08.180695 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:08.180672    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f31be4a0-295f-4ed0-901d-f45cee943496-istio-envoy\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk\" (UID: \"f31be4a0-295f-4ed0-901d-f45cee943496\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk"
Oct 30 13:15:08.181109 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:08.181087    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f31be4a0-295f-4ed0-901d-f45cee943496-istio-podinfo\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk\" (UID: \"f31be4a0-295f-4ed0-901d-f45cee943496\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk"
Oct 30 13:15:08.200679 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:08.200617    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f31be4a0-295f-4ed0-901d-f45cee943496-istio-token\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk\" (UID: \"f31be4a0-295f-4ed0-901d-f45cee943496\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk"
Oct 30 13:15:08.200842 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:08.200690    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hsms\" (UniqueName: \"kubernetes.io/projected/f31be4a0-295f-4ed0-901d-f45cee943496-kube-api-access-9hsms\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk\" (UID: \"f31be4a0-295f-4ed0-901d-f45cee943496\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk"
Oct 30 13:15:08.376048 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:08.375950    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk"
Oct 30 13:15:08.696653 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:15:08.696605    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf31be4a0_295f_4ed0_901d_f45cee943496.slice/crio-4a53e2945cda948c91a194a60e64d665686213e085aa11b39f0c4955445833ff WatchSource:0}: Error finding container 4a53e2945cda948c91a194a60e64d665686213e085aa11b39f0c4955445833ff: Status 404 returned error can't find the container with id 4a53e2945cda948c91a194a60e64d665686213e085aa11b39f0c4955445833ff
Oct 30 13:15:08.697598 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:08.697572    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk"]
Oct 30 13:15:08.699797 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:08.699764    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:15:08.699877 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:08.699846    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:15:08.699915 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:08.699885    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:15:08.734443 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:08.734404    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk" event={"ID":"f31be4a0-295f-4ed0-901d-f45cee943496","Type":"ContainerStarted","Data":"4a53e2945cda948c91a194a60e64d665686213e085aa11b39f0c4955445833ff"}
Oct 30 13:15:09.755154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:09.754801    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk" event={"ID":"f31be4a0-295f-4ed0-901d-f45cee943496","Type":"ContainerStarted","Data":"e7b90ddc2fd7f3e171c51a7c9a784840409d6376fc3f6f62a5de5b27e85725ad"}
Oct 30 13:15:09.837909 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:09.837842    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk" podStartSLOduration=2.837819883 podStartE2EDuration="2.837819883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:09.813122582 +0000 UTC m=+728.960389817" watchObservedRunningTime="2026-10-30 13:15:09.837819883 +0000 UTC m=+728.985087122"
Oct 30 13:15:10.376556 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:10.376523    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk"
Oct 30 13:15:10.378494 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:10.378473    5093 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 13:15:10.378558 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:10.378521    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk" podUID="f31be4a0-295f-4ed0-901d-f45cee943496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 13:15:11.376628 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:11.376592    5093 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 13:15:11.377157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:11.376656    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk" podUID="f31be4a0-295f-4ed0-901d-f45cee943496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 13:15:12.377358 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:12.377307    5093 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 13:15:12.377906 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:12.377372    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk" podUID="f31be4a0-295f-4ed0-901d-f45cee943496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 13:15:13.377821 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:13.377247    5093 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 13:15:13.377821 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:13.377323    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk" podUID="f31be4a0-295f-4ed0-901d-f45cee943496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 13:15:14.376547 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:14.376510    5093 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 13:15:14.376747 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:14.376573    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk" podUID="f31be4a0-295f-4ed0-901d-f45cee943496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 13:15:15.527763 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:15.527728    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk"
Oct 30 13:15:15.528731 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:15.528700    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk"
Oct 30 13:15:15.529882 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:15.529857    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk"
Oct 30 13:15:21.167421 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:15:21.167382    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:15:31.329027 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:31.328991    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5fbdd899c5-g6rrt"]
Oct 30 13:15:31.333938 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:31.333910    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5fbdd899c5-g6rrt"
Oct 30 13:15:31.351255 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:31.351225    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5fbdd899c5-g6rrt"]
Oct 30 13:15:31.360922 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:31.360893    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-7827742ffd45\""
Oct 30 13:15:31.361080 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:31.360953    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-132ab67347e8\""
Oct 30 13:15:31.361080 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:31.360968    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-1856bd8d477d\""
Oct 30 13:15:31.361080 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:31.361047    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-b6xtq\""
Oct 30 13:15:31.361080 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:31.361053    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 13:15:31.361337 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:31.360912    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-cf7e1905f255\""
Oct 30 13:15:31.364489 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:31.364465    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 13:15:31.364616 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:31.364505    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 13:15:31.364616 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:31.364515    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 13:15:31.447825 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:31.447780    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj27b\" (UniqueName: \"kubernetes.io/projected/e7facca3-a73c-4fb6-873b-efdc2a4300c5-kube-api-access-jj27b\") pod \"payload-processing-external-model-5fbdd899c5-g6rrt\" (UID: \"e7facca3-a73c-4fb6-873b-efdc2a4300c5\") " pod="models-as-a-service/payload-processing-external-model-5fbdd899c5-g6rrt"
Oct 30 13:15:31.448049 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:31.447945    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e7facca3-a73c-4fb6-873b-efdc2a4300c5-routing-overlay-volume\") pod \"payload-processing-external-model-5fbdd899c5-g6rrt\" (UID: \"e7facca3-a73c-4fb6-873b-efdc2a4300c5\") " pod="models-as-a-service/payload-processing-external-model-5fbdd899c5-g6rrt"
Oct 30 13:15:31.448049 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:31.447995    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e7facca3-a73c-4fb6-873b-efdc2a4300c5-provider-credentials\") pod \"payload-processing-external-model-5fbdd899c5-g6rrt\" (UID: \"e7facca3-a73c-4fb6-873b-efdc2a4300c5\") " pod="models-as-a-service/payload-processing-external-model-5fbdd899c5-g6rrt"
Oct 30 13:15:31.448191 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:31.448100    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7facca3-a73c-4fb6-873b-efdc2a4300c5-plugins-config-volume\") pod \"payload-processing-external-model-5fbdd899c5-g6rrt\" (UID: \"e7facca3-a73c-4fb6-873b-efdc2a4300c5\") " pod="models-as-a-service/payload-processing-external-model-5fbdd899c5-g6rrt"
Oct 30 13:15:31.549032 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:31.548991    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e7facca3-a73c-4fb6-873b-efdc2a4300c5-routing-overlay-volume\") pod \"payload-processing-external-model-5fbdd899c5-g6rrt\" (UID: \"e7facca3-a73c-4fb6-873b-efdc2a4300c5\") " pod="models-as-a-service/payload-processing-external-model-5fbdd899c5-g6rrt"
Oct 30 13:15:31.549259 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:31.549054    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e7facca3-a73c-4fb6-873b-efdc2a4300c5-provider-credentials\") pod \"payload-processing-external-model-5fbdd899c5-g6rrt\" (UID: \"e7facca3-a73c-4fb6-873b-efdc2a4300c5\") " pod="models-as-a-service/payload-processing-external-model-5fbdd899c5-g6rrt"
Oct 30 13:15:31.549259 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:31.549112    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7facca3-a73c-4fb6-873b-efdc2a4300c5-plugins-config-volume\") pod \"payload-processing-external-model-5fbdd899c5-g6rrt\" (UID: \"e7facca3-a73c-4fb6-873b-efdc2a4300c5\") " pod="models-as-a-service/payload-processing-external-model-5fbdd899c5-g6rrt"
Oct 30 13:15:31.549259 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:31.549205    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj27b\" (UniqueName: \"kubernetes.io/projected/e7facca3-a73c-4fb6-873b-efdc2a4300c5-kube-api-access-jj27b\") pod \"payload-processing-external-model-5fbdd899c5-g6rrt\" (UID: \"e7facca3-a73c-4fb6-873b-efdc2a4300c5\") " pod="models-as-a-service/payload-processing-external-model-5fbdd899c5-g6rrt"
Oct 30 13:15:31.549753 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:31.549732    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e7facca3-a73c-4fb6-873b-efdc2a4300c5-routing-overlay-volume\") pod \"payload-processing-external-model-5fbdd899c5-g6rrt\" (UID: \"e7facca3-a73c-4fb6-873b-efdc2a4300c5\") " pod="models-as-a-service/payload-processing-external-model-5fbdd899c5-g6rrt"
Oct 30 13:15:31.550006 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:31.549980    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7facca3-a73c-4fb6-873b-efdc2a4300c5-plugins-config-volume\") pod \"payload-processing-external-model-5fbdd899c5-g6rrt\" (UID: \"e7facca3-a73c-4fb6-873b-efdc2a4300c5\") " pod="models-as-a-service/payload-processing-external-model-5fbdd899c5-g6rrt"
Oct 30 13:15:31.551866 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:31.551843    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e7facca3-a73c-4fb6-873b-efdc2a4300c5-provider-credentials\") pod \"payload-processing-external-model-5fbdd899c5-g6rrt\" (UID: \"e7facca3-a73c-4fb6-873b-efdc2a4300c5\") " pod="models-as-a-service/payload-processing-external-model-5fbdd899c5-g6rrt"
Oct 30 13:15:31.569959 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:31.569936    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj27b\" (UniqueName: \"kubernetes.io/projected/e7facca3-a73c-4fb6-873b-efdc2a4300c5-kube-api-access-jj27b\") pod \"payload-processing-external-model-5fbdd899c5-g6rrt\" (UID: \"e7facca3-a73c-4fb6-873b-efdc2a4300c5\") " pod="models-as-a-service/payload-processing-external-model-5fbdd899c5-g6rrt"
Oct 30 13:15:31.651525 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:31.651423    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5fbdd899c5-g6rrt"
Oct 30 13:15:31.949560 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:31.949529    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5fbdd899c5-g6rrt"]
Oct 30 13:15:31.949682 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:15:31.949549    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7facca3_a73c_4fb6_873b_efdc2a4300c5.slice/crio-ee7058ff6a4b54ed79addb6f0b5a2df46113870d73125945cbd6be28d75d6db8 WatchSource:0}: Error finding container ee7058ff6a4b54ed79addb6f0b5a2df46113870d73125945cbd6be28d75d6db8: Status 404 returned error can't find the container with id ee7058ff6a4b54ed79addb6f0b5a2df46113870d73125945cbd6be28d75d6db8
Oct 30 13:15:32.956922 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:32.956884    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5fbdd899c5-g6rrt" event={"ID":"e7facca3-a73c-4fb6-873b-efdc2a4300c5","Type":"ContainerStarted","Data":"c3a695d0b9d6e81a18547efad72ab3aede14cc6c8fb52a9e6a7ded0e4a662764"}
Oct 30 13:15:32.957397 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:32.956937    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5fbdd899c5-g6rrt" event={"ID":"e7facca3-a73c-4fb6-873b-efdc2a4300c5","Type":"ContainerStarted","Data":"ee7058ff6a4b54ed79addb6f0b5a2df46113870d73125945cbd6be28d75d6db8"}
Oct 30 13:15:32.957397 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:32.957025    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5fbdd899c5-g6rrt"
Oct 30 13:15:32.999667 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:32.999619    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5fbdd899c5-g6rrt" podStartSLOduration=1.9996062540000001 podStartE2EDuration="1.999606254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:32.999167318 +0000 UTC m=+752.146434556" watchObservedRunningTime="2026-10-30 13:15:32.999606254 +0000 UTC m=+752.146873496"
Oct 30 13:15:35.724378 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:35.724341    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7"]
Oct 30 13:15:35.728698 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:35.728670    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7"
Oct 30 13:15:35.772399 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:35.772365    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7"]
Oct 30 13:15:35.791745 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:35.791706    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24a8f884-84d1-4852-80d1-fe699060a90a-istiod-ca-cert\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7\" (UID: \"24a8f884-84d1-4852-80d1-fe699060a90a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7"
Oct 30 13:15:35.791917 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:35.791765    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a8f884-84d1-4852-80d1-fe699060a90a-workload-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7\" (UID: \"24a8f884-84d1-4852-80d1-fe699060a90a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7"
Oct 30 13:15:35.791917 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:35.791855    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24a8f884-84d1-4852-80d1-fe699060a90a-istio-podinfo\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7\" (UID: \"24a8f884-84d1-4852-80d1-fe699060a90a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7"
Oct 30 13:15:35.791917 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:35.791902    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24a8f884-84d1-4852-80d1-fe699060a90a-istio-data\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7\" (UID: \"24a8f884-84d1-4852-80d1-fe699060a90a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7"
Oct 30 13:15:35.792036 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:35.791965    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktbpk\" (UniqueName: \"kubernetes.io/projected/24a8f884-84d1-4852-80d1-fe699060a90a-kube-api-access-ktbpk\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7\" (UID: \"24a8f884-84d1-4852-80d1-fe699060a90a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7"
Oct 30 13:15:35.792036 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:35.792000    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24a8f884-84d1-4852-80d1-fe699060a90a-workload-certs\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7\" (UID: \"24a8f884-84d1-4852-80d1-fe699060a90a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7"
Oct 30 13:15:35.792036 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:35.792015    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24a8f884-84d1-4852-80d1-fe699060a90a-istio-token\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7\" (UID: \"24a8f884-84d1-4852-80d1-fe699060a90a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7"
Oct 30 13:15:35.792161 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:35.792043    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24a8f884-84d1-4852-80d1-fe699060a90a-istio-envoy\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7\" (UID: \"24a8f884-84d1-4852-80d1-fe699060a90a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7"
Oct 30 13:15:35.792161 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:35.792119    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a8f884-84d1-4852-80d1-fe699060a90a-credential-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7\" (UID: \"24a8f884-84d1-4852-80d1-fe699060a90a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7"
Oct 30 13:15:35.813025 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:35.812997    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-5852ff-openshift-default-dockercfg-76pkk\""
Oct 30 13:15:35.892646 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:35.892605    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24a8f884-84d1-4852-80d1-fe699060a90a-istio-data\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7\" (UID: \"24a8f884-84d1-4852-80d1-fe699060a90a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7"
Oct 30 13:15:35.892820 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:35.892681    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktbpk\" (UniqueName: \"kubernetes.io/projected/24a8f884-84d1-4852-80d1-fe699060a90a-kube-api-access-ktbpk\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7\" (UID: \"24a8f884-84d1-4852-80d1-fe699060a90a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7"
Oct 30 13:15:35.892820 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:35.892725    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24a8f884-84d1-4852-80d1-fe699060a90a-workload-certs\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7\" (UID: \"24a8f884-84d1-4852-80d1-fe699060a90a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7"
Oct 30 13:15:35.892820 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:35.892751    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24a8f884-84d1-4852-80d1-fe699060a90a-istio-token\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7\" (UID: \"24a8f884-84d1-4852-80d1-fe699060a90a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7"
Oct 30 13:15:35.892820 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:35.892797    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24a8f884-84d1-4852-80d1-fe699060a90a-istio-envoy\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7\" (UID: \"24a8f884-84d1-4852-80d1-fe699060a90a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7"
Oct 30 13:15:35.892976 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:35.892943    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a8f884-84d1-4852-80d1-fe699060a90a-credential-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7\" (UID: \"24a8f884-84d1-4852-80d1-fe699060a90a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7"
Oct 30 13:15:35.893068 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:35.893048    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24a8f884-84d1-4852-80d1-fe699060a90a-istiod-ca-cert\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7\" (UID: \"24a8f884-84d1-4852-80d1-fe699060a90a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7"
Oct 30 13:15:35.893171 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:35.893064    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24a8f884-84d1-4852-80d1-fe699060a90a-istio-data\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7\" (UID: \"24a8f884-84d1-4852-80d1-fe699060a90a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7"
Oct 30 13:15:35.893171 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:35.893095    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24a8f884-84d1-4852-80d1-fe699060a90a-workload-certs\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7\" (UID: \"24a8f884-84d1-4852-80d1-fe699060a90a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7"
Oct 30 13:15:35.893171 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:35.893098    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a8f884-84d1-4852-80d1-fe699060a90a-workload-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7\" (UID: \"24a8f884-84d1-4852-80d1-fe699060a90a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7"
Oct 30 13:15:35.893348 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:35.893201    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24a8f884-84d1-4852-80d1-fe699060a90a-istio-podinfo\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7\" (UID: \"24a8f884-84d1-4852-80d1-fe699060a90a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7"
Oct 30 13:15:35.893348 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:35.893329    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a8f884-84d1-4852-80d1-fe699060a90a-credential-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7\" (UID: \"24a8f884-84d1-4852-80d1-fe699060a90a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7"
Oct 30 13:15:35.893423 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:35.893406    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a8f884-84d1-4852-80d1-fe699060a90a-workload-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7\" (UID: \"24a8f884-84d1-4852-80d1-fe699060a90a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7"
Oct 30 13:15:35.893660 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:35.893639    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24a8f884-84d1-4852-80d1-fe699060a90a-istiod-ca-cert\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7\" (UID: \"24a8f884-84d1-4852-80d1-fe699060a90a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7"
Oct 30 13:15:35.895086 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:35.895059    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24a8f884-84d1-4852-80d1-fe699060a90a-istio-envoy\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7\" (UID: \"24a8f884-84d1-4852-80d1-fe699060a90a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7"
Oct 30 13:15:35.895483 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:35.895465    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24a8f884-84d1-4852-80d1-fe699060a90a-istio-podinfo\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7\" (UID: \"24a8f884-84d1-4852-80d1-fe699060a90a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7"
Oct 30 13:15:35.913560 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:35.913533    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24a8f884-84d1-4852-80d1-fe699060a90a-istio-token\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7\" (UID: \"24a8f884-84d1-4852-80d1-fe699060a90a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7"
Oct 30 13:15:35.914110 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:35.914092    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktbpk\" (UniqueName: \"kubernetes.io/projected/24a8f884-84d1-4852-80d1-fe699060a90a-kube-api-access-ktbpk\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7\" (UID: \"24a8f884-84d1-4852-80d1-fe699060a90a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7"
Oct 30 13:15:36.043457 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:36.043358    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7"
Oct 30 13:15:36.358097 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:15:36.358057    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24a8f884_84d1_4852_80d1_fe699060a90a.slice/crio-78e3554182cbb7a5f689e13628fd2eb3e746188592baf3731f31f042814f3f80 WatchSource:0}: Error finding container 78e3554182cbb7a5f689e13628fd2eb3e746188592baf3731f31f042814f3f80: Status 404 returned error can't find the container with id 78e3554182cbb7a5f689e13628fd2eb3e746188592baf3731f31f042814f3f80
Oct 30 13:15:36.359994 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:36.359923    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7"]
Oct 30 13:15:36.361390 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:36.361355    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:15:36.361505 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:36.361433    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:15:36.361505 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:36.361482    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:15:37.024376 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:37.024331    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7" event={"ID":"24a8f884-84d1-4852-80d1-fe699060a90a","Type":"ContainerStarted","Data":"12bcb869687f7f2f28e6c5f696a4a3da8e7df9a9b0988d6cb88aab99b5bd1f87"}
Oct 30 13:15:37.024978 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:37.024408    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7" event={"ID":"24a8f884-84d1-4852-80d1-fe699060a90a","Type":"ContainerStarted","Data":"78e3554182cbb7a5f689e13628fd2eb3e746188592baf3731f31f042814f3f80"}
Oct 30 13:15:37.046513 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:37.046480    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7"
Oct 30 13:15:37.067312 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:37.067267    5093 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 13:15:37.110416 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:37.110369    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7" podUID="24a8f884-84d1-4852-80d1-fe699060a90a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 13:15:37.130154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:37.121839    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7" podStartSLOduration=2.12181761 podStartE2EDuration="2.12181761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:37.101183494 +0000 UTC m=+756.248450736" watchObservedRunningTime="2026-10-30 13:15:37.12181761 +0000 UTC m=+756.269084852"
Oct 30 13:15:38.051156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:38.048516    5093 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 13:15:38.051156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:38.048581    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7" podUID="24a8f884-84d1-4852-80d1-fe699060a90a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 13:15:39.061155 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:39.044687    5093 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 13:15:39.061155 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:39.044745    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7" podUID="24a8f884-84d1-4852-80d1-fe699060a90a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 13:15:39.688154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:39.681311    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-knngs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:15:39.688154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:39.681384    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs" podUID="c417d75a-c5bf-42fa-8aff-335a63c1295e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:15:40.051213 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:40.050621    5093 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 13:15:40.051213 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:40.050718    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7" podUID="24a8f884-84d1-4852-80d1-fe699060a90a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 13:15:40.103157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:40.102271    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx" podUID="ba156eeb-9955-4f9a-857b-2c4d502a5fcc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 30 13:15:40.398218 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:40.397231    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv" podUID="d6a44981-104f-4d3e-9937-d184a6ddc482" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:15:41.056195 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:41.050526    5093 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 13:15:41.056195 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:41.050592    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7" podUID="24a8f884-84d1-4852-80d1-fe699060a90a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 13:15:42.050631 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:42.050580    5093 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 13:15:42.051208 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:42.050657    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7" podUID="24a8f884-84d1-4852-80d1-fe699060a90a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 13:15:43.044124 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:43.044075    5093 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 13:15:43.044337 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:43.044169    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7" podUID="24a8f884-84d1-4852-80d1-fe699060a90a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 13:15:43.967164 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:43.967116    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5fbdd899c5-g6rrt"
Oct 30 13:15:44.596701 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:44.596668    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7"
Oct 30 13:15:44.597085 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:44.597058    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7"
Oct 30 13:15:44.597911 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:15:44.597882    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7"
Oct 30 13:15:51.184808 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:15:51.184759    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:16:11.128376 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:11.128335    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw"]
Oct 30 13:16:11.133369 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:11.133340    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw"
Oct 30 13:16:11.149484 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:11.149454    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw"]
Oct 30 13:16:11.158876 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:11.158847    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-52af49\"/\"default-dockercfg-x88rn\""
Oct 30 13:16:11.162473 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:11.162447    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-52af49\"/\"kube-root-ca.crt\""
Oct 30 13:16:11.162473 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:11.162463    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-52af49\"/\"distinct-w1-52af49-kserve-self-signed-certs\""
Oct 30 13:16:11.162675 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:11.162579    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-52af49\"/\"openshift-service-ca.crt\""
Oct 30 13:16:11.252512 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:11.252475    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad37954e-1350-4f75-a63f-a0c0f29ae416-tmp-dir\") pod \"distinct-w1-52af49-kserve-6d595fc8c-88lfw\" (UID: \"ad37954e-1350-4f75-a63f-a0c0f29ae416\") " pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw"
Oct 30 13:16:11.252688 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:11.252526    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmzbf\" (UniqueName: \"kubernetes.io/projected/ad37954e-1350-4f75-a63f-a0c0f29ae416-kube-api-access-qmzbf\") pod \"distinct-w1-52af49-kserve-6d595fc8c-88lfw\" (UID: \"ad37954e-1350-4f75-a63f-a0c0f29ae416\") " pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw"
Oct 30 13:16:11.252688 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:11.252641    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad37954e-1350-4f75-a63f-a0c0f29ae416-model-cache\") pod \"distinct-w1-52af49-kserve-6d595fc8c-88lfw\" (UID: \"ad37954e-1350-4f75-a63f-a0c0f29ae416\") " pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw"
Oct 30 13:16:11.252776 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:11.252743    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad37954e-1350-4f75-a63f-a0c0f29ae416-tls-certs\") pod \"distinct-w1-52af49-kserve-6d595fc8c-88lfw\" (UID: \"ad37954e-1350-4f75-a63f-a0c0f29ae416\") " pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw"
Oct 30 13:16:11.252820 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:11.252792    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad37954e-1350-4f75-a63f-a0c0f29ae416-home\") pod \"distinct-w1-52af49-kserve-6d595fc8c-88lfw\" (UID: \"ad37954e-1350-4f75-a63f-a0c0f29ae416\") " pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw"
Oct 30 13:16:11.252820 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:11.252812    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad37954e-1350-4f75-a63f-a0c0f29ae416-dshm\") pod \"distinct-w1-52af49-kserve-6d595fc8c-88lfw\" (UID: \"ad37954e-1350-4f75-a63f-a0c0f29ae416\") " pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw"
Oct 30 13:16:11.353531 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:11.353492    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad37954e-1350-4f75-a63f-a0c0f29ae416-home\") pod \"distinct-w1-52af49-kserve-6d595fc8c-88lfw\" (UID: \"ad37954e-1350-4f75-a63f-a0c0f29ae416\") " pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw"
Oct 30 13:16:11.353732 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:11.353543    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad37954e-1350-4f75-a63f-a0c0f29ae416-dshm\") pod \"distinct-w1-52af49-kserve-6d595fc8c-88lfw\" (UID: \"ad37954e-1350-4f75-a63f-a0c0f29ae416\") " pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw"
Oct 30 13:16:11.353732 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:11.353645    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad37954e-1350-4f75-a63f-a0c0f29ae416-tmp-dir\") pod \"distinct-w1-52af49-kserve-6d595fc8c-88lfw\" (UID: \"ad37954e-1350-4f75-a63f-a0c0f29ae416\") " pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw"
Oct 30 13:16:11.353732 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:11.353682    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmzbf\" (UniqueName: \"kubernetes.io/projected/ad37954e-1350-4f75-a63f-a0c0f29ae416-kube-api-access-qmzbf\") pod \"distinct-w1-52af49-kserve-6d595fc8c-88lfw\" (UID: \"ad37954e-1350-4f75-a63f-a0c0f29ae416\") " pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw"
Oct 30 13:16:11.353928 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:11.353737    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad37954e-1350-4f75-a63f-a0c0f29ae416-model-cache\") pod \"distinct-w1-52af49-kserve-6d595fc8c-88lfw\" (UID: \"ad37954e-1350-4f75-a63f-a0c0f29ae416\") " pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw"
Oct 30 13:16:11.353928 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:11.353822    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad37954e-1350-4f75-a63f-a0c0f29ae416-tls-certs\") pod \"distinct-w1-52af49-kserve-6d595fc8c-88lfw\" (UID: \"ad37954e-1350-4f75-a63f-a0c0f29ae416\") " pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw"
Oct 30 13:16:11.354045 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:11.353940    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad37954e-1350-4f75-a63f-a0c0f29ae416-home\") pod \"distinct-w1-52af49-kserve-6d595fc8c-88lfw\" (UID: \"ad37954e-1350-4f75-a63f-a0c0f29ae416\") " pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw"
Oct 30 13:16:11.354414 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:11.354383    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad37954e-1350-4f75-a63f-a0c0f29ae416-model-cache\") pod \"distinct-w1-52af49-kserve-6d595fc8c-88lfw\" (UID: \"ad37954e-1350-4f75-a63f-a0c0f29ae416\") " pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw"
Oct 30 13:16:11.354554 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:11.354461    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad37954e-1350-4f75-a63f-a0c0f29ae416-tmp-dir\") pod \"distinct-w1-52af49-kserve-6d595fc8c-88lfw\" (UID: \"ad37954e-1350-4f75-a63f-a0c0f29ae416\") " pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw"
Oct 30 13:16:11.356593 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:11.356570    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad37954e-1350-4f75-a63f-a0c0f29ae416-dshm\") pod \"distinct-w1-52af49-kserve-6d595fc8c-88lfw\" (UID: \"ad37954e-1350-4f75-a63f-a0c0f29ae416\") " pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw"
Oct 30 13:16:11.368777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:11.368746    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad37954e-1350-4f75-a63f-a0c0f29ae416-tls-certs\") pod \"distinct-w1-52af49-kserve-6d595fc8c-88lfw\" (UID: \"ad37954e-1350-4f75-a63f-a0c0f29ae416\") " pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw"
Oct 30 13:16:11.374124 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:11.374098    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmzbf\" (UniqueName: \"kubernetes.io/projected/ad37954e-1350-4f75-a63f-a0c0f29ae416-kube-api-access-qmzbf\") pod \"distinct-w1-52af49-kserve-6d595fc8c-88lfw\" (UID: \"ad37954e-1350-4f75-a63f-a0c0f29ae416\") " pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw"
Oct 30 13:16:11.448543 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:11.448505    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw"
Oct 30 13:16:11.762584 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:11.762549    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw"]
Oct 30 13:16:11.763426 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:16:11.763396    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad37954e_1350_4f75_a63f_a0c0f29ae416.slice/crio-24d2a0d50321fd7da922b39d6d28a9aae8fcf07e8734c9eda1a151c7372d8db6 WatchSource:0}: Error finding container 24d2a0d50321fd7da922b39d6d28a9aae8fcf07e8734c9eda1a151c7372d8db6: Status 404 returned error can't find the container with id 24d2a0d50321fd7da922b39d6d28a9aae8fcf07e8734c9eda1a151c7372d8db6
Oct 30 13:16:12.440626 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:12.440582    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw" event={"ID":"ad37954e-1350-4f75-a63f-a0c0f29ae416","Type":"ContainerStarted","Data":"eee06b34a6dfc140bceba2e50b20760552420543366306b80dd1580e7e56c366"}
Oct 30 13:16:12.441065 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:12.440641    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw" event={"ID":"ad37954e-1350-4f75-a63f-a0c0f29ae416","Type":"ContainerStarted","Data":"24d2a0d50321fd7da922b39d6d28a9aae8fcf07e8734c9eda1a151c7372d8db6"}
Oct 30 13:16:12.448914 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:12.448887    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw"
Oct 30 13:16:12.485970 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:12.485919    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw" podStartSLOduration=1.485905174 podStartE2EDuration="1.485905174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:16:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:16:12.482662015 +0000 UTC m=+791.629929255" watchObservedRunningTime="2026-10-30 13:16:12.485905174 +0000 UTC m=+791.633172415"
Oct 30 13:16:20.862262 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:20.862221    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq"]
Oct 30 13:16:20.866948 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:20.866922    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq"
Oct 30 13:16:20.896878 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:20.896841    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq"]
Oct 30 13:16:20.963734 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:20.963698    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a79305d1-bc2a-448f-be51-58affd67dd8a-istio-data\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq\" (UID: \"a79305d1-bc2a-448f-be51-58affd67dd8a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq"
Oct 30 13:16:20.963955 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:20.963910    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mhf8\" (UniqueName: \"kubernetes.io/projected/a79305d1-bc2a-448f-be51-58affd67dd8a-kube-api-access-7mhf8\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq\" (UID: \"a79305d1-bc2a-448f-be51-58affd67dd8a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq"
Oct 30 13:16:20.963955 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:20.963945    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a79305d1-bc2a-448f-be51-58affd67dd8a-istiod-ca-cert\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq\" (UID: \"a79305d1-bc2a-448f-be51-58affd67dd8a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq"
Oct 30 13:16:20.964085 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:20.964056    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79305d1-bc2a-448f-be51-58affd67dd8a-workload-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq\" (UID: \"a79305d1-bc2a-448f-be51-58affd67dd8a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq"
Oct 30 13:16:20.964122 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:20.964098    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a79305d1-bc2a-448f-be51-58affd67dd8a-istio-token\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq\" (UID: \"a79305d1-bc2a-448f-be51-58affd67dd8a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq"
Oct 30 13:16:20.964186 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:20.964154    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79305d1-bc2a-448f-be51-58affd67dd8a-credential-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq\" (UID: \"a79305d1-bc2a-448f-be51-58affd67dd8a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq"
Oct 30 13:16:20.964242 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:20.964229    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a79305d1-bc2a-448f-be51-58affd67dd8a-workload-certs\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq\" (UID: \"a79305d1-bc2a-448f-be51-58affd67dd8a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq"
Oct 30 13:16:20.964319 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:20.964308    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a79305d1-bc2a-448f-be51-58affd67dd8a-istio-podinfo\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq\" (UID: \"a79305d1-bc2a-448f-be51-58affd67dd8a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq"
Oct 30 13:16:20.964386 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:20.964372    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a79305d1-bc2a-448f-be51-58affd67dd8a-istio-envoy\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq\" (UID: \"a79305d1-bc2a-448f-be51-58affd67dd8a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq"
Oct 30 13:16:21.065110 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:21.065072    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mhf8\" (UniqueName: \"kubernetes.io/projected/a79305d1-bc2a-448f-be51-58affd67dd8a-kube-api-access-7mhf8\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq\" (UID: \"a79305d1-bc2a-448f-be51-58affd67dd8a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq"
Oct 30 13:16:21.065307 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:21.065116    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a79305d1-bc2a-448f-be51-58affd67dd8a-istiod-ca-cert\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq\" (UID: \"a79305d1-bc2a-448f-be51-58affd67dd8a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq"
Oct 30 13:16:21.065307 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:21.065182    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79305d1-bc2a-448f-be51-58affd67dd8a-workload-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq\" (UID: \"a79305d1-bc2a-448f-be51-58affd67dd8a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq"
Oct 30 13:16:21.065307 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:21.065208    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a79305d1-bc2a-448f-be51-58affd67dd8a-istio-token\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq\" (UID: \"a79305d1-bc2a-448f-be51-58affd67dd8a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq"
Oct 30 13:16:21.065307 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:21.065276    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79305d1-bc2a-448f-be51-58affd67dd8a-credential-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq\" (UID: \"a79305d1-bc2a-448f-be51-58affd67dd8a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq"
Oct 30 13:16:21.065556 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:21.065313    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a79305d1-bc2a-448f-be51-58affd67dd8a-workload-certs\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq\" (UID: \"a79305d1-bc2a-448f-be51-58affd67dd8a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq"
Oct 30 13:16:21.065556 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:21.065443    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a79305d1-bc2a-448f-be51-58affd67dd8a-istio-podinfo\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq\" (UID: \"a79305d1-bc2a-448f-be51-58affd67dd8a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq"
Oct 30 13:16:21.065556 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:21.065490    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a79305d1-bc2a-448f-be51-58affd67dd8a-istio-envoy\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq\" (UID: \"a79305d1-bc2a-448f-be51-58affd67dd8a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq"
Oct 30 13:16:21.065556 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:21.065529    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a79305d1-bc2a-448f-be51-58affd67dd8a-istio-data\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq\" (UID: \"a79305d1-bc2a-448f-be51-58affd67dd8a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq"
Oct 30 13:16:21.065770 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:21.065657    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79305d1-bc2a-448f-be51-58affd67dd8a-workload-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq\" (UID: \"a79305d1-bc2a-448f-be51-58affd67dd8a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq"
Oct 30 13:16:21.065770 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:21.065702    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79305d1-bc2a-448f-be51-58affd67dd8a-credential-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq\" (UID: \"a79305d1-bc2a-448f-be51-58affd67dd8a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq"
Oct 30 13:16:21.065899 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:21.065872    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a79305d1-bc2a-448f-be51-58affd67dd8a-istio-data\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq\" (UID: \"a79305d1-bc2a-448f-be51-58affd67dd8a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq"
Oct 30 13:16:21.066336 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:21.066311    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a79305d1-bc2a-448f-be51-58affd67dd8a-workload-certs\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq\" (UID: \"a79305d1-bc2a-448f-be51-58affd67dd8a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq"
Oct 30 13:16:21.066510 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:21.066332    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a79305d1-bc2a-448f-be51-58affd67dd8a-istiod-ca-cert\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq\" (UID: \"a79305d1-bc2a-448f-be51-58affd67dd8a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq"
Oct 30 13:16:21.069658 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:21.069631    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a79305d1-bc2a-448f-be51-58affd67dd8a-istio-podinfo\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq\" (UID: \"a79305d1-bc2a-448f-be51-58affd67dd8a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq"
Oct 30 13:16:21.071381 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:21.071356    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a79305d1-bc2a-448f-be51-58affd67dd8a-istio-envoy\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq\" (UID: \"a79305d1-bc2a-448f-be51-58affd67dd8a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq"
Oct 30 13:16:21.084871 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:21.084833    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5fbdd899c5-g6rrt"]
Oct 30 13:16:21.092424 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:21.091837    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a79305d1-bc2a-448f-be51-58affd67dd8a-istio-token\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq\" (UID: \"a79305d1-bc2a-448f-be51-58affd67dd8a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq"
Oct 30 13:16:21.092424 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:21.092085    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mhf8\" (UniqueName: \"kubernetes.io/projected/a79305d1-bc2a-448f-be51-58affd67dd8a-kube-api-access-7mhf8\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq\" (UID: \"a79305d1-bc2a-448f-be51-58affd67dd8a\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq"
Oct 30 13:16:21.172046 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:16:21.172013    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:16:21.183873 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:21.182439    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq"
Oct 30 13:16:21.448994 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:21.448896    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw"
Oct 30 13:16:21.462327 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:21.462290    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw"
Oct 30 13:16:21.535178 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:21.535121    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq"]
Oct 30 13:16:21.537750 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:21.537710    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:16:21.537875 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:21.537854    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:16:21.537989 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:21.537896    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:16:21.548575 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:21.548553    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw"
Oct 30 13:16:22.548155 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:22.547412    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq" event={"ID":"a79305d1-bc2a-448f-be51-58affd67dd8a","Type":"ContainerStarted","Data":"f725306152f7cf8d7dc58e3d87719ab4eb14639642021784d881db300352b48a"}
Oct 30 13:16:22.548155 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:22.547473    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq" event={"ID":"a79305d1-bc2a-448f-be51-58affd67dd8a","Type":"ContainerStarted","Data":"6937527fef8f398531a125fc1347a5868dfbb7599748c609252ce3efd48275f3"}
Oct 30 13:16:22.622121 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:22.622064    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq" podStartSLOduration=2.6220466030000003 podStartE2EDuration="2.622046603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:16:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:16:22.606818807 +0000 UTC m=+801.754086074" watchObservedRunningTime="2026-10-30 13:16:22.622046603 +0000 UTC m=+801.769313844"
Oct 30 13:16:23.183179 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:23.183123    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq"
Oct 30 13:16:23.184512 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:23.184483    5093 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 13:16:23.184669 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:23.184545    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq" podUID="a79305d1-bc2a-448f-be51-58affd67dd8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 13:16:24.183834 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:24.183793    5093 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 13:16:24.184285 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:24.183852    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq" podUID="a79305d1-bc2a-448f-be51-58affd67dd8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 13:16:25.183441 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:25.183404    5093 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 13:16:25.183621 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:25.183466    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq" podUID="a79305d1-bc2a-448f-be51-58affd67dd8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 13:16:26.085395 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:26.085343    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5fbdd899c5-g6rrt" podUID="e7facca3-a73c-4fb6-873b-efdc2a4300c5" containerName="payload-processing" containerID="cri-o://c3a695d0b9d6e81a18547efad72ab3aede14cc6c8fb52a9e6a7ded0e4a662764" gracePeriod=25
Oct 30 13:16:26.183883 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:26.183847    5093 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 13:16:26.184091 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:26.183942    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq" podUID="a79305d1-bc2a-448f-be51-58affd67dd8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 13:16:26.621152 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:26.618014    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5fbdd899c5-g6rrt"
Oct 30 13:16:26.621152 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:26.619247    5093 generic.go:356] "Generic (PLEG): container finished" podID="e7facca3-a73c-4fb6-873b-efdc2a4300c5" containerID="c3a695d0b9d6e81a18547efad72ab3aede14cc6c8fb52a9e6a7ded0e4a662764" exitCode=0
Oct 30 13:16:26.621152 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:26.619482    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5fbdd899c5-g6rrt" event={"ID":"e7facca3-a73c-4fb6-873b-efdc2a4300c5","Type":"ContainerDied","Data":"c3a695d0b9d6e81a18547efad72ab3aede14cc6c8fb52a9e6a7ded0e4a662764"}
Oct 30 13:16:26.621152 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:26.619535    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5fbdd899c5-g6rrt" event={"ID":"e7facca3-a73c-4fb6-873b-efdc2a4300c5","Type":"ContainerDied","Data":"ee7058ff6a4b54ed79addb6f0b5a2df46113870d73125945cbd6be28d75d6db8"}
Oct 30 13:16:26.621152 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:26.619564    5093 scope.go:122] "RemoveContainer" containerID="c3a695d0b9d6e81a18547efad72ab3aede14cc6c8fb52a9e6a7ded0e4a662764"
Oct 30 13:16:26.650907 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:26.650878    5093 scope.go:122] "RemoveContainer" containerID="c3a695d0b9d6e81a18547efad72ab3aede14cc6c8fb52a9e6a7ded0e4a662764"
Oct 30 13:16:26.651618 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:16:26.651552    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3a695d0b9d6e81a18547efad72ab3aede14cc6c8fb52a9e6a7ded0e4a662764\": container with ID starting with c3a695d0b9d6e81a18547efad72ab3aede14cc6c8fb52a9e6a7ded0e4a662764 not found: ID does not exist" containerID="c3a695d0b9d6e81a18547efad72ab3aede14cc6c8fb52a9e6a7ded0e4a662764"
Oct 30 13:16:26.651618 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:26.651589    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3a695d0b9d6e81a18547efad72ab3aede14cc6c8fb52a9e6a7ded0e4a662764"} err="failed to get container status \"c3a695d0b9d6e81a18547efad72ab3aede14cc6c8fb52a9e6a7ded0e4a662764\": rpc error: code = NotFound desc = could not find container \"c3a695d0b9d6e81a18547efad72ab3aede14cc6c8fb52a9e6a7ded0e4a662764\": container with ID starting with c3a695d0b9d6e81a18547efad72ab3aede14cc6c8fb52a9e6a7ded0e4a662764 not found: ID does not exist"
Oct 30 13:16:26.708862 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:26.706600    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7facca3-a73c-4fb6-873b-efdc2a4300c5-kube-api-access-jj27b\" (UniqueName: \"kubernetes.io/projected/e7facca3-a73c-4fb6-873b-efdc2a4300c5-kube-api-access-jj27b\") pod \"e7facca3-a73c-4fb6-873b-efdc2a4300c5\" (UID: \"e7facca3-a73c-4fb6-873b-efdc2a4300c5\") "
Oct 30 13:16:26.708862 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:26.706687    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7facca3-a73c-4fb6-873b-efdc2a4300c5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e7facca3-a73c-4fb6-873b-efdc2a4300c5-routing-overlay-volume\") pod \"e7facca3-a73c-4fb6-873b-efdc2a4300c5\" (UID: \"e7facca3-a73c-4fb6-873b-efdc2a4300c5\") "
Oct 30 13:16:26.708862 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:26.706735    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7facca3-a73c-4fb6-873b-efdc2a4300c5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e7facca3-a73c-4fb6-873b-efdc2a4300c5-provider-credentials\") pod \"e7facca3-a73c-4fb6-873b-efdc2a4300c5\" (UID: \"e7facca3-a73c-4fb6-873b-efdc2a4300c5\") "
Oct 30 13:16:26.708862 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:26.706772    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7facca3-a73c-4fb6-873b-efdc2a4300c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7facca3-a73c-4fb6-873b-efdc2a4300c5-plugins-config-volume\") pod \"e7facca3-a73c-4fb6-873b-efdc2a4300c5\" (UID: \"e7facca3-a73c-4fb6-873b-efdc2a4300c5\") "
Oct 30 13:16:26.724184 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:26.721620    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7facca3-a73c-4fb6-873b-efdc2a4300c5-plugins-config-volume" pod "e7facca3-a73c-4fb6-873b-efdc2a4300c5" (UID: "e7facca3-a73c-4fb6-873b-efdc2a4300c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:16:26.724726 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:26.724669    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7facca3-a73c-4fb6-873b-efdc2a4300c5-routing-overlay-volume" pod "e7facca3-a73c-4fb6-873b-efdc2a4300c5" (UID: "e7facca3-a73c-4fb6-873b-efdc2a4300c5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:16:26.808248 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:26.808106    5093 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e7facca3-a73c-4fb6-873b-efdc2a4300c5-routing-overlay-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:16:26.808248 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:26.808162    5093 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7facca3-a73c-4fb6-873b-efdc2a4300c5-plugins-config-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:16:26.823785 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:26.823728    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7facca3-a73c-4fb6-873b-efdc2a4300c5-kube-api-access-jj27b" pod "e7facca3-a73c-4fb6-873b-efdc2a4300c5" (UID: "e7facca3-a73c-4fb6-873b-efdc2a4300c5"). InnerVolumeSpecName "kube-api-access-jj27b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:16:26.898247 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:26.898184    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7facca3-a73c-4fb6-873b-efdc2a4300c5-provider-credentials" pod "e7facca3-a73c-4fb6-873b-efdc2a4300c5" (UID: "e7facca3-a73c-4fb6-873b-efdc2a4300c5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:16:26.911494 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:26.909145    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj27b\" (UniqueName: \"kubernetes.io/projected/e7facca3-a73c-4fb6-873b-efdc2a4300c5-kube-api-access-jj27b\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:16:26.911494 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:26.909184    5093 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e7facca3-a73c-4fb6-873b-efdc2a4300c5-provider-credentials\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:16:27.216150 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:27.196526    5093 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 13:16:27.216150 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:27.214203    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq" podUID="a79305d1-bc2a-448f-be51-58affd67dd8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 13:16:27.658196 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:27.658161    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5fbdd899c5-g6rrt"
Oct 30 13:16:27.720049 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:27.720011    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5fbdd899c5-g6rrt"]
Oct 30 13:16:27.757358 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:27.757321    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5fbdd899c5-g6rrt"]
Oct 30 13:16:28.183108 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:28.183071    5093 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 13:16:28.183330 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:28.183150    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq" podUID="a79305d1-bc2a-448f-be51-58affd67dd8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 13:16:28.310041 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:28.309938    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td"]
Oct 30 13:16:28.310656 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:28.310636    5093 cpu_manager.go:414] "containerMap: removing container" podUID="e7facca3-a73c-4fb6-873b-efdc2a4300c5" containerName="payload-processing"
Oct 30 13:16:28.310656 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:28.310656    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7facca3-a73c-4fb6-873b-efdc2a4300c5" containerName="payload-processing"
Oct 30 13:16:28.310845 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:28.310755    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7facca3-a73c-4fb6-873b-efdc2a4300c5" containerName="payload-processing"
Oct 30 13:16:28.315683 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:28.315660    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td"
Oct 30 13:16:28.345758 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:28.345649    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9bf8ad\"/\"e2e-conflict-b-dfdb6918-kserve-self-signed-certs\""
Oct 30 13:16:28.350752 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:28.350711    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td"]
Oct 30 13:16:28.355648 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:28.355601    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c64ad75a-3364-408f-b79c-46781ff2745c-model-cache\") pod \"e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td\" (UID: \"c64ad75a-3364-408f-b79c-46781ff2745c\") " pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td"
Oct 30 13:16:28.355818 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:28.355656    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c64ad75a-3364-408f-b79c-46781ff2745c-home\") pod \"e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td\" (UID: \"c64ad75a-3364-408f-b79c-46781ff2745c\") " pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td"
Oct 30 13:16:28.356040 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:28.356010    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c64ad75a-3364-408f-b79c-46781ff2745c-tls-certs\") pod \"e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td\" (UID: \"c64ad75a-3364-408f-b79c-46781ff2745c\") " pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td"
Oct 30 13:16:28.356171 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:28.356119    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c64ad75a-3364-408f-b79c-46781ff2745c-dshm\") pod \"e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td\" (UID: \"c64ad75a-3364-408f-b79c-46781ff2745c\") " pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td"
Oct 30 13:16:28.356258 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:28.356206    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69v27\" (UniqueName: \"kubernetes.io/projected/c64ad75a-3364-408f-b79c-46781ff2745c-kube-api-access-69v27\") pod \"e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td\" (UID: \"c64ad75a-3364-408f-b79c-46781ff2745c\") " pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td"
Oct 30 13:16:28.356258 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:28.356241    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c64ad75a-3364-408f-b79c-46781ff2745c-tmp-dir\") pod \"e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td\" (UID: \"c64ad75a-3364-408f-b79c-46781ff2745c\") " pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td"
Oct 30 13:16:28.457866 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:28.457760    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c64ad75a-3364-408f-b79c-46781ff2745c-model-cache\") pod \"e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td\" (UID: \"c64ad75a-3364-408f-b79c-46781ff2745c\") " pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td"
Oct 30 13:16:28.457866 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:28.457817    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c64ad75a-3364-408f-b79c-46781ff2745c-home\") pod \"e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td\" (UID: \"c64ad75a-3364-408f-b79c-46781ff2745c\") " pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td"
Oct 30 13:16:28.458110 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:28.457876    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c64ad75a-3364-408f-b79c-46781ff2745c-tls-certs\") pod \"e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td\" (UID: \"c64ad75a-3364-408f-b79c-46781ff2745c\") " pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td"
Oct 30 13:16:28.458110 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:28.457928    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c64ad75a-3364-408f-b79c-46781ff2745c-dshm\") pod \"e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td\" (UID: \"c64ad75a-3364-408f-b79c-46781ff2745c\") " pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td"
Oct 30 13:16:28.458110 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:28.457993    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69v27\" (UniqueName: \"kubernetes.io/projected/c64ad75a-3364-408f-b79c-46781ff2745c-kube-api-access-69v27\") pod \"e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td\" (UID: \"c64ad75a-3364-408f-b79c-46781ff2745c\") " pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td"
Oct 30 13:16:28.458110 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:28.458033    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c64ad75a-3364-408f-b79c-46781ff2745c-tmp-dir\") pod \"e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td\" (UID: \"c64ad75a-3364-408f-b79c-46781ff2745c\") " pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td"
Oct 30 13:16:28.458369 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:28.458293    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c64ad75a-3364-408f-b79c-46781ff2745c-model-cache\") pod \"e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td\" (UID: \"c64ad75a-3364-408f-b79c-46781ff2745c\") " pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td"
Oct 30 13:16:28.458369 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:28.458309    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c64ad75a-3364-408f-b79c-46781ff2745c-home\") pod \"e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td\" (UID: \"c64ad75a-3364-408f-b79c-46781ff2745c\") " pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td"
Oct 30 13:16:28.458607 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:28.458583    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c64ad75a-3364-408f-b79c-46781ff2745c-tmp-dir\") pod \"e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td\" (UID: \"c64ad75a-3364-408f-b79c-46781ff2745c\") " pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td"
Oct 30 13:16:28.460479 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:28.460453    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c64ad75a-3364-408f-b79c-46781ff2745c-dshm\") pod \"e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td\" (UID: \"c64ad75a-3364-408f-b79c-46781ff2745c\") " pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td"
Oct 30 13:16:28.460577 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:28.460559    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c64ad75a-3364-408f-b79c-46781ff2745c-tls-certs\") pod \"e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td\" (UID: \"c64ad75a-3364-408f-b79c-46781ff2745c\") " pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td"
Oct 30 13:16:28.489161 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:28.482596    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69v27\" (UniqueName: \"kubernetes.io/projected/c64ad75a-3364-408f-b79c-46781ff2745c-kube-api-access-69v27\") pod \"e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td\" (UID: \"c64ad75a-3364-408f-b79c-46781ff2745c\") " pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td"
Oct 30 13:16:28.630289 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:28.630250    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td"
Oct 30 13:16:28.944359 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:28.944310    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td"]
Oct 30 13:16:29.001120 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:29.001087    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7facca3-a73c-4fb6-873b-efdc2a4300c5" path="/var/lib/kubelet/pods/e7facca3-a73c-4fb6-873b-efdc2a4300c5/volumes"
Oct 30 13:16:29.673813 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:29.673777    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td" event={"ID":"c64ad75a-3364-408f-b79c-46781ff2745c","Type":"ContainerStarted","Data":"6f112ceed82707e9f2664d4e3d8c56e16c33dcc14eddd74ba05b16da93320781"}
Oct 30 13:16:29.674365 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:29.673828    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td" event={"ID":"c64ad75a-3364-408f-b79c-46781ff2745c","Type":"ContainerStarted","Data":"bf75388f12b8375a98b703d10f1f937bbce037b3aec03b24593eb4c78ed77683"}
Oct 30 13:16:29.715383 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:29.715327    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td" podStartSLOduration=1.7153117679999998 podStartE2EDuration="1.715311768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:16:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:16:29.714865617 +0000 UTC m=+808.862132858" watchObservedRunningTime="2026-10-30 13:16:29.715311768 +0000 UTC m=+808.862579009"
Oct 30 13:16:30.225293 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:30.225246    5093 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:16:30.225512 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:30.225315    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq" podUID="a79305d1-bc2a-448f-be51-58affd67dd8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:16:30.474586 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:30.474548    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq"
Oct 30 13:16:30.475009 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:30.474982    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq"
Oct 30 13:16:30.475740 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:30.475675    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq"
Oct 30 13:16:30.631448 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:30.631402    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td"
Oct 30 13:16:35.913784 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:35.913740    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g"]
Oct 30 13:16:35.918246 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:35.918216    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g"
Oct 30 13:16:35.960818 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:35.960778    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g"]
Oct 30 13:16:36.062696 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:36.062663    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/911b7cb8-3eac-4576-be8b-3fecd0fea186-istio-token\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g\" (UID: \"911b7cb8-3eac-4576-be8b-3fecd0fea186\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g"
Oct 30 13:16:36.062696 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:36.062710    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/911b7cb8-3eac-4576-be8b-3fecd0fea186-credential-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g\" (UID: \"911b7cb8-3eac-4576-be8b-3fecd0fea186\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g"
Oct 30 13:16:36.062963 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:36.062739    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/911b7cb8-3eac-4576-be8b-3fecd0fea186-workload-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g\" (UID: \"911b7cb8-3eac-4576-be8b-3fecd0fea186\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g"
Oct 30 13:16:36.062963 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:36.062928    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27qvn\" (UniqueName: \"kubernetes.io/projected/911b7cb8-3eac-4576-be8b-3fecd0fea186-kube-api-access-27qvn\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g\" (UID: \"911b7cb8-3eac-4576-be8b-3fecd0fea186\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g"
Oct 30 13:16:36.063098 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:36.063020    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/911b7cb8-3eac-4576-be8b-3fecd0fea186-istiod-ca-cert\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g\" (UID: \"911b7cb8-3eac-4576-be8b-3fecd0fea186\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g"
Oct 30 13:16:36.063098 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:36.063076    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/911b7cb8-3eac-4576-be8b-3fecd0fea186-workload-certs\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g\" (UID: \"911b7cb8-3eac-4576-be8b-3fecd0fea186\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g"
Oct 30 13:16:36.063284 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:36.063264    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/911b7cb8-3eac-4576-be8b-3fecd0fea186-istio-podinfo\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g\" (UID: \"911b7cb8-3eac-4576-be8b-3fecd0fea186\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g"
Oct 30 13:16:36.063360 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:36.063305    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/911b7cb8-3eac-4576-be8b-3fecd0fea186-istio-envoy\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g\" (UID: \"911b7cb8-3eac-4576-be8b-3fecd0fea186\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g"
Oct 30 13:16:36.063423 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:36.063410    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/911b7cb8-3eac-4576-be8b-3fecd0fea186-istio-data\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g\" (UID: \"911b7cb8-3eac-4576-be8b-3fecd0fea186\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g"
Oct 30 13:16:36.164996 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:36.164895    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/911b7cb8-3eac-4576-be8b-3fecd0fea186-istio-data\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g\" (UID: \"911b7cb8-3eac-4576-be8b-3fecd0fea186\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g"
Oct 30 13:16:36.165188 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:36.165072    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/911b7cb8-3eac-4576-be8b-3fecd0fea186-istio-token\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g\" (UID: \"911b7cb8-3eac-4576-be8b-3fecd0fea186\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g"
Oct 30 13:16:36.165188 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:36.165121    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/911b7cb8-3eac-4576-be8b-3fecd0fea186-credential-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g\" (UID: \"911b7cb8-3eac-4576-be8b-3fecd0fea186\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g"
Oct 30 13:16:36.165331 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:36.165203    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/911b7cb8-3eac-4576-be8b-3fecd0fea186-workload-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g\" (UID: \"911b7cb8-3eac-4576-be8b-3fecd0fea186\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g"
Oct 30 13:16:36.165331 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:36.165235    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27qvn\" (UniqueName: \"kubernetes.io/projected/911b7cb8-3eac-4576-be8b-3fecd0fea186-kube-api-access-27qvn\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g\" (UID: \"911b7cb8-3eac-4576-be8b-3fecd0fea186\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g"
Oct 30 13:16:36.165331 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:36.165264    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/911b7cb8-3eac-4576-be8b-3fecd0fea186-istiod-ca-cert\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g\" (UID: \"911b7cb8-3eac-4576-be8b-3fecd0fea186\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g"
Oct 30 13:16:36.165331 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:36.165291    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/911b7cb8-3eac-4576-be8b-3fecd0fea186-workload-certs\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g\" (UID: \"911b7cb8-3eac-4576-be8b-3fecd0fea186\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g"
Oct 30 13:16:36.165571 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:36.165382    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/911b7cb8-3eac-4576-be8b-3fecd0fea186-istio-data\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g\" (UID: \"911b7cb8-3eac-4576-be8b-3fecd0fea186\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g"
Oct 30 13:16:36.165571 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:36.165401    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/911b7cb8-3eac-4576-be8b-3fecd0fea186-istio-podinfo\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g\" (UID: \"911b7cb8-3eac-4576-be8b-3fecd0fea186\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g"
Oct 30 13:16:36.165571 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:36.165455    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/911b7cb8-3eac-4576-be8b-3fecd0fea186-istio-envoy\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g\" (UID: \"911b7cb8-3eac-4576-be8b-3fecd0fea186\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g"
Oct 30 13:16:36.165571 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:36.165559    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/911b7cb8-3eac-4576-be8b-3fecd0fea186-credential-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g\" (UID: \"911b7cb8-3eac-4576-be8b-3fecd0fea186\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g"
Oct 30 13:16:36.165717 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:36.165583    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/911b7cb8-3eac-4576-be8b-3fecd0fea186-workload-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g\" (UID: \"911b7cb8-3eac-4576-be8b-3fecd0fea186\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g"
Oct 30 13:16:36.166050 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:36.166020    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/911b7cb8-3eac-4576-be8b-3fecd0fea186-workload-certs\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g\" (UID: \"911b7cb8-3eac-4576-be8b-3fecd0fea186\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g"
Oct 30 13:16:36.166413 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:36.166386    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/911b7cb8-3eac-4576-be8b-3fecd0fea186-istiod-ca-cert\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g\" (UID: \"911b7cb8-3eac-4576-be8b-3fecd0fea186\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g"
Oct 30 13:16:36.167779 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:36.167754    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/911b7cb8-3eac-4576-be8b-3fecd0fea186-istio-envoy\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g\" (UID: \"911b7cb8-3eac-4576-be8b-3fecd0fea186\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g"
Oct 30 13:16:36.168012 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:36.167995    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/911b7cb8-3eac-4576-be8b-3fecd0fea186-istio-podinfo\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g\" (UID: \"911b7cb8-3eac-4576-be8b-3fecd0fea186\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g"
Oct 30 13:16:36.209481 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:36.209287    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/911b7cb8-3eac-4576-be8b-3fecd0fea186-istio-token\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g\" (UID: \"911b7cb8-3eac-4576-be8b-3fecd0fea186\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g"
Oct 30 13:16:36.209655 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:36.209503    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27qvn\" (UniqueName: \"kubernetes.io/projected/911b7cb8-3eac-4576-be8b-3fecd0fea186-kube-api-access-27qvn\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g\" (UID: \"911b7cb8-3eac-4576-be8b-3fecd0fea186\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g"
Oct 30 13:16:36.235603 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:36.235566    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g"
Oct 30 13:16:36.584209 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:36.584181    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g"]
Oct 30 13:16:36.587530 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:16:36.587492    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod911b7cb8_3eac_4576_be8b_3fecd0fea186.slice/crio-5ea11e8a9aac1d511e7ea004ed9c15edcd718fc97361dd3c6f9dee7e3d44e7f7 WatchSource:0}: Error finding container 5ea11e8a9aac1d511e7ea004ed9c15edcd718fc97361dd3c6f9dee7e3d44e7f7: Status 404 returned error can't find the container with id 5ea11e8a9aac1d511e7ea004ed9c15edcd718fc97361dd3c6f9dee7e3d44e7f7
Oct 30 13:16:36.594191 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:36.591278    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:16:36.594191 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:36.591375    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:16:36.594191 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:36.591417    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:16:36.773217 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:36.769818    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g" event={"ID":"911b7cb8-3eac-4576-be8b-3fecd0fea186","Type":"ContainerStarted","Data":"b91f94f69b5ab993381a08fcfcf0ad8f0745fc5627a050c0285a8160cd8ba288"}
Oct 30 13:16:36.773217 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:36.769936    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g" event={"ID":"911b7cb8-3eac-4576-be8b-3fecd0fea186","Type":"ContainerStarted","Data":"5ea11e8a9aac1d511e7ea004ed9c15edcd718fc97361dd3c6f9dee7e3d44e7f7"}
Oct 30 13:16:37.238595 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:37.238559    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g"
Oct 30 13:16:37.239377 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:37.239351    5093 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 13:16:37.239568 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:37.239542    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g" podUID="911b7cb8-3eac-4576-be8b-3fecd0fea186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 13:16:37.963165 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:37.961077    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g" podStartSLOduration=2.961052573 podStartE2EDuration="2.961052573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:16:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:16:36.840986419 +0000 UTC m=+815.988253657" watchObservedRunningTime="2026-10-30 13:16:37.961052573 +0000 UTC m=+817.108319816"
Oct 30 13:16:38.006218 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:38.006179    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn"]
Oct 30 13:16:38.027473 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:38.027440    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn"
Oct 30 13:16:38.058160 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:38.050483    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-52af49-openshift-default-dockercfg-tbzp6\""
Oct 30 13:16:38.068165 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:38.059254    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn"]
Oct 30 13:16:38.165154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:38.143355    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwvph\" (UniqueName: \"kubernetes.io/projected/de03af69-2ff9-4add-a940-740935198076-kube-api-access-kwvph\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn\" (UID: \"de03af69-2ff9-4add-a940-740935198076\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn"
Oct 30 13:16:38.171006 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:38.170972    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de03af69-2ff9-4add-a940-740935198076-istio-envoy\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn\" (UID: \"de03af69-2ff9-4add-a940-740935198076\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn"
Oct 30 13:16:38.171286 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:38.171262    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de03af69-2ff9-4add-a940-740935198076-workload-certs\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn\" (UID: \"de03af69-2ff9-4add-a940-740935198076\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn"
Oct 30 13:16:38.171524 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:38.171504    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de03af69-2ff9-4add-a940-740935198076-credential-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn\" (UID: \"de03af69-2ff9-4add-a940-740935198076\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn"
Oct 30 13:16:38.171655 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:38.171638    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de03af69-2ff9-4add-a940-740935198076-workload-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn\" (UID: \"de03af69-2ff9-4add-a940-740935198076\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn"
Oct 30 13:16:38.182154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:38.171797    5093 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx" podUID="ba156eeb-9955-4f9a-857b-2c4d502a5fcc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:16:38.182154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:38.172101    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de03af69-2ff9-4add-a940-740935198076-istio-token\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn\" (UID: \"de03af69-2ff9-4add-a940-740935198076\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn"
Oct 30 13:16:38.182154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:38.172159    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de03af69-2ff9-4add-a940-740935198076-istio-data\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn\" (UID: \"de03af69-2ff9-4add-a940-740935198076\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn"
Oct 30 13:16:38.182154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:38.172288    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de03af69-2ff9-4add-a940-740935198076-istio-podinfo\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn\" (UID: \"de03af69-2ff9-4add-a940-740935198076\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn"
Oct 30 13:16:38.182154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:38.172361    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de03af69-2ff9-4add-a940-740935198076-istiod-ca-cert\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn\" (UID: \"de03af69-2ff9-4add-a940-740935198076\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn"
Oct 30 13:16:38.240213 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:38.224880    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt" podUID="4ae52c69-0c30-438f-9ac0-c87b6de16061" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:16:38.302394 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:38.239602    5093 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 13:16:38.313162 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:38.308532    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g" podUID="911b7cb8-3eac-4576-be8b-3fecd0fea186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 13:16:38.313566 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:38.283742    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de03af69-2ff9-4add-a940-740935198076-istio-envoy\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn\" (UID: \"de03af69-2ff9-4add-a940-740935198076\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn"
Oct 30 13:16:38.355253 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:38.355213    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de03af69-2ff9-4add-a940-740935198076-workload-certs\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn\" (UID: \"de03af69-2ff9-4add-a940-740935198076\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn"
Oct 30 13:16:38.356797 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:38.356768    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de03af69-2ff9-4add-a940-740935198076-credential-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn\" (UID: \"de03af69-2ff9-4add-a940-740935198076\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn"
Oct 30 13:16:38.379243 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:38.379211    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de03af69-2ff9-4add-a940-740935198076-workload-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn\" (UID: \"de03af69-2ff9-4add-a940-740935198076\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn"
Oct 30 13:16:38.380223 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:38.380197    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de03af69-2ff9-4add-a940-740935198076-istio-token\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn\" (UID: \"de03af69-2ff9-4add-a940-740935198076\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn"
Oct 30 13:16:38.380406 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:38.380390    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de03af69-2ff9-4add-a940-740935198076-istio-data\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn\" (UID: \"de03af69-2ff9-4add-a940-740935198076\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn"
Oct 30 13:16:38.380610 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:38.380594    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de03af69-2ff9-4add-a940-740935198076-istio-podinfo\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn\" (UID: \"de03af69-2ff9-4add-a940-740935198076\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn"
Oct 30 13:16:38.380771 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:38.380756    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de03af69-2ff9-4add-a940-740935198076-istiod-ca-cert\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn\" (UID: \"de03af69-2ff9-4add-a940-740935198076\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn"
Oct 30 13:16:38.380938 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:38.380923    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwvph\" (UniqueName: \"kubernetes.io/projected/de03af69-2ff9-4add-a940-740935198076-kube-api-access-kwvph\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn\" (UID: \"de03af69-2ff9-4add-a940-740935198076\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn"
Oct 30 13:16:38.381372 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:38.379802    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de03af69-2ff9-4add-a940-740935198076-credential-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn\" (UID: \"de03af69-2ff9-4add-a940-740935198076\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn"
Oct 30 13:16:38.387203 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:38.379854    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv" podUID="d6a44981-104f-4d3e-9937-d184a6ddc482" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:16:38.387671 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:38.356646    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de03af69-2ff9-4add-a940-740935198076-workload-certs\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn\" (UID: \"de03af69-2ff9-4add-a940-740935198076\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn"
Oct 30 13:16:38.387799 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:38.382235    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de03af69-2ff9-4add-a940-740935198076-istio-data\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn\" (UID: \"de03af69-2ff9-4add-a940-740935198076\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn"
Oct 30 13:16:38.387892 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:38.380107    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de03af69-2ff9-4add-a940-740935198076-workload-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn\" (UID: \"de03af69-2ff9-4add-a940-740935198076\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn"
Oct 30 13:16:38.388303 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:38.388281    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de03af69-2ff9-4add-a940-740935198076-istiod-ca-cert\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn\" (UID: \"de03af69-2ff9-4add-a940-740935198076\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn"
Oct 30 13:16:38.461066 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:38.461028    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de03af69-2ff9-4add-a940-740935198076-istio-envoy\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn\" (UID: \"de03af69-2ff9-4add-a940-740935198076\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn"
Oct 30 13:16:38.475509 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:38.463339    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de03af69-2ff9-4add-a940-740935198076-istio-podinfo\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn\" (UID: \"de03af69-2ff9-4add-a940-740935198076\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn"
Oct 30 13:16:38.494862 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:38.494782    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de03af69-2ff9-4add-a940-740935198076-istio-token\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn\" (UID: \"de03af69-2ff9-4add-a940-740935198076\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn"
Oct 30 13:16:38.529986 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:38.529945    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwvph\" (UniqueName: \"kubernetes.io/projected/de03af69-2ff9-4add-a940-740935198076-kube-api-access-kwvph\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn\" (UID: \"de03af69-2ff9-4add-a940-740935198076\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn"
Oct 30 13:16:38.592172 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:38.592109    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw" podUID="ad37954e-1350-4f75-a63f-a0c0f29ae416" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:16:38.616395 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:38.616357    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn"
Oct 30 13:16:38.705567 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:38.704699    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td"
Oct 30 13:16:38.808079 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:38.807970    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg" podUID="ea079ac1-f502-4b5f-b16a-c91e2de98de8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:16:39.041051 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:39.041017    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td"
Oct 30 13:16:39.126364 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:39.124705    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx" podUID="ba156eeb-9955-4f9a-857b-2c4d502a5fcc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:16:39.242164 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:39.241557    5093 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 13:16:39.242164 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:39.241624    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g" podUID="911b7cb8-3eac-4576-be8b-3fecd0fea186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 13:16:39.312274 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:39.312222    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt" podUID="4ae52c69-0c30-438f-9ac0-c87b6de16061" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 30 13:16:39.313168 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:39.313112    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt"
Oct 30 13:16:39.686437 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:39.685206    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv" podUID="d6a44981-104f-4d3e-9937-d184a6ddc482" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 13:16:39.686437 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:39.685643    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv"
Oct 30 13:16:39.699014 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:39.687328    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn"]
Oct 30 13:16:39.699014 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:39.687625    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw" podUID="ad37954e-1350-4f75-a63f-a0c0f29ae416" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 13:16:39.699014 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:39.688646    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw"
Oct 30 13:16:39.757391 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:39.755945    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-knngs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:16:39.757391 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:39.756001    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs" podUID="c417d75a-c5bf-42fa-8aff-335a63c1295e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:16:39.786908 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:39.786863    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:16:39.787192 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:39.787165    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:16:39.787354 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:39.787333    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:16:39.838946 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:39.838900    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg" podUID="ea079ac1-f502-4b5f-b16a-c91e2de98de8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:16:39.846619 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:39.846587    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg"
Oct 30 13:16:40.020590 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:40.020543    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn" event={"ID":"de03af69-2ff9-4add-a940-740935198076","Type":"ContainerStarted","Data":"1e4f6faed3c5e30e4d0c2a068b0c5829786d7b817fcb7a48e0a583703835f058"}
Oct 30 13:16:40.126408 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:40.125239    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx" podUID="ba156eeb-9955-4f9a-857b-2c4d502a5fcc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:16:40.128591 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:40.127107    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx"
Oct 30 13:16:40.128591 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:40.128227    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td" podUID="c64ad75a-3364-408f-b79c-46781ff2745c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:16:40.313190 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:40.312876    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt" podUID="4ae52c69-0c30-438f-9ac0-c87b6de16061" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 30 13:16:40.428900 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:40.428865    5093 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 13:16:40.429213 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:40.429167    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g" podUID="911b7cb8-3eac-4576-be8b-3fecd0fea186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 13:16:40.474737 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:40.474681    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8269v" podUID="3d487ab7-4654-4757-ae7c-4250db7f2ef5" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.38:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:16:40.614318 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:40.614220    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td"
Oct 30 13:16:40.673981 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:40.673945    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx"
Oct 30 13:16:40.694978 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:40.694928    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw" podUID="ad37954e-1350-4f75-a63f-a0c0f29ae416" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 13:16:40.716568 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:40.707342    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg"
Oct 30 13:16:40.716568 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:40.715098    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv" podUID="d6a44981-104f-4d3e-9937-d184a6ddc482" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:16:40.781467 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:40.781335    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt"
Oct 30 13:16:40.787618 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:40.787577    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw"
Oct 30 13:16:40.805662 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:40.805627    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv"
Oct 30 13:16:41.036836 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:41.036790    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn" event={"ID":"de03af69-2ff9-4add-a940-740935198076","Type":"ContainerStarted","Data":"10a47553747aa9b3ccf7bf837fbdaaeeeeb1935e4daa11719b3564f6782ef83a"}
Oct 30 13:16:41.101853 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:41.101793    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn" podStartSLOduration=4.101778148 podStartE2EDuration="4.101778148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:16:41.097067783 +0000 UTC m=+820.244335024" watchObservedRunningTime="2026-10-30 13:16:41.101778148 +0000 UTC m=+820.249045445"
Oct 30 13:16:41.236584 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:41.236545    5093 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 13:16:41.236880 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:41.236853    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g" podUID="911b7cb8-3eac-4576-be8b-3fecd0fea186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 13:16:41.617376 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:41.617333    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn"
Oct 30 13:16:41.618618 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:41.618593    5093 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 13:16:41.618735 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:41.618644    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn" podUID="de03af69-2ff9-4add-a940-740935198076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 13:16:42.236581 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:42.236540    5093 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 13:16:42.236888 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:42.236858    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g" podUID="911b7cb8-3eac-4576-be8b-3fecd0fea186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 13:16:42.617710 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:42.617626    5093 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 13:16:42.618200 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:42.618173    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn" podUID="de03af69-2ff9-4add-a940-740935198076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 13:16:42.886548 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:42.886429    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc"]
Oct 30 13:16:42.891145 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:42.891092    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc"
Oct 30 13:16:42.904842 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:42.904803    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc"]
Oct 30 13:16:42.913464 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:42.913432    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-f9ebb5-openshift-default-dockercfg-259p4\""
Oct 30 13:16:43.000923 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:43.000875    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvk4j\" (UniqueName: \"kubernetes.io/projected/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-kube-api-access-vvk4j\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc\" (UID: \"8daf18c1-014e-4c6b-bffb-bb25ceef69f3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc"
Oct 30 13:16:43.001277 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:43.001249    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-istio-podinfo\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc\" (UID: \"8daf18c1-014e-4c6b-bffb-bb25ceef69f3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc"
Oct 30 13:16:43.001491 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:43.001472    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-workload-certs\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc\" (UID: \"8daf18c1-014e-4c6b-bffb-bb25ceef69f3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc"
Oct 30 13:16:43.001664 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:43.001648    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-istio-token\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc\" (UID: \"8daf18c1-014e-4c6b-bffb-bb25ceef69f3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc"
Oct 30 13:16:43.001788 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:43.001774    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-istio-data\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc\" (UID: \"8daf18c1-014e-4c6b-bffb-bb25ceef69f3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc"
Oct 30 13:16:43.001894 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:43.001880    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-istiod-ca-cert\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc\" (UID: \"8daf18c1-014e-4c6b-bffb-bb25ceef69f3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc"
Oct 30 13:16:43.002017 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:43.002003    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-credential-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc\" (UID: \"8daf18c1-014e-4c6b-bffb-bb25ceef69f3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc"
Oct 30 13:16:43.002170 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:43.002150    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-workload-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc\" (UID: \"8daf18c1-014e-4c6b-bffb-bb25ceef69f3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc"
Oct 30 13:16:43.002300 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:43.002282    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-istio-envoy\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc\" (UID: \"8daf18c1-014e-4c6b-bffb-bb25ceef69f3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc"
Oct 30 13:16:43.103902 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:43.103864    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-credential-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc\" (UID: \"8daf18c1-014e-4c6b-bffb-bb25ceef69f3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc"
Oct 30 13:16:43.103902 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:43.103904    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-workload-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc\" (UID: \"8daf18c1-014e-4c6b-bffb-bb25ceef69f3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc"
Oct 30 13:16:43.104195 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:43.103921    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-istio-envoy\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc\" (UID: \"8daf18c1-014e-4c6b-bffb-bb25ceef69f3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc"
Oct 30 13:16:43.104195 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:43.103972    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvk4j\" (UniqueName: \"kubernetes.io/projected/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-kube-api-access-vvk4j\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc\" (UID: \"8daf18c1-014e-4c6b-bffb-bb25ceef69f3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc"
Oct 30 13:16:43.104195 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:43.104000    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-istio-podinfo\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc\" (UID: \"8daf18c1-014e-4c6b-bffb-bb25ceef69f3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc"
Oct 30 13:16:43.104195 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:43.104055    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-workload-certs\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc\" (UID: \"8daf18c1-014e-4c6b-bffb-bb25ceef69f3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc"
Oct 30 13:16:43.104195 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:43.104111    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-istio-token\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc\" (UID: \"8daf18c1-014e-4c6b-bffb-bb25ceef69f3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc"
Oct 30 13:16:43.104195 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:43.104156    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-istio-data\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc\" (UID: \"8daf18c1-014e-4c6b-bffb-bb25ceef69f3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc"
Oct 30 13:16:43.104195 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:43.104174    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-istiod-ca-cert\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc\" (UID: \"8daf18c1-014e-4c6b-bffb-bb25ceef69f3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc"
Oct 30 13:16:43.104540 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:43.104342    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-credential-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc\" (UID: \"8daf18c1-014e-4c6b-bffb-bb25ceef69f3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc"
Oct 30 13:16:43.104813 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:43.104790    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-istiod-ca-cert\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc\" (UID: \"8daf18c1-014e-4c6b-bffb-bb25ceef69f3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc"
Oct 30 13:16:43.104914 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:43.104813    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-workload-certs\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc\" (UID: \"8daf18c1-014e-4c6b-bffb-bb25ceef69f3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc"
Oct 30 13:16:43.105225 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:43.105185    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-workload-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc\" (UID: \"8daf18c1-014e-4c6b-bffb-bb25ceef69f3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc"
Oct 30 13:16:43.105361 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:43.105271    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-istio-data\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc\" (UID: \"8daf18c1-014e-4c6b-bffb-bb25ceef69f3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc"
Oct 30 13:16:43.106411 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:43.106391    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-istio-envoy\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc\" (UID: \"8daf18c1-014e-4c6b-bffb-bb25ceef69f3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc"
Oct 30 13:16:43.106872 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:43.106850    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-istio-podinfo\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc\" (UID: \"8daf18c1-014e-4c6b-bffb-bb25ceef69f3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc"
Oct 30 13:16:43.146215 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:43.145694    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-istio-token\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc\" (UID: \"8daf18c1-014e-4c6b-bffb-bb25ceef69f3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc"
Oct 30 13:16:43.164732 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:43.161566    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvk4j\" (UniqueName: \"kubernetes.io/projected/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-kube-api-access-vvk4j\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc\" (UID: \"8daf18c1-014e-4c6b-bffb-bb25ceef69f3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc"
Oct 30 13:16:43.207277 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:43.207122    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc"
Oct 30 13:16:43.249162 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:43.236533    5093 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 13:16:43.249162 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:43.236603    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g" podUID="911b7cb8-3eac-4576-be8b-3fecd0fea186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 13:16:43.617442 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:43.617409    5093 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 13:16:43.617610 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:43.617466    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn" podUID="de03af69-2ff9-4add-a940-740935198076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 13:16:43.675409 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:16:43.673826    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8daf18c1_014e_4c6b_bffb_bb25ceef69f3.slice/crio-2701fd856b5045dd9505a9b6fa015cbd4270105b6869939175eae5a1239fc600 WatchSource:0}: Error finding container 2701fd856b5045dd9505a9b6fa015cbd4270105b6869939175eae5a1239fc600: Status 404 returned error can't find the container with id 2701fd856b5045dd9505a9b6fa015cbd4270105b6869939175eae5a1239fc600
Oct 30 13:16:43.677739 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:43.677709    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc"]
Oct 30 13:16:43.682464 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:43.682433    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:16:43.682575 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:43.682498    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:16:43.682575 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:43.682528    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:16:44.108588 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:44.108545    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc" event={"ID":"8daf18c1-014e-4c6b-bffb-bb25ceef69f3","Type":"ContainerStarted","Data":"9598582f32e50cedf06d82de1ba545385878ca286d586f3a05e582bedab4db9f"}
Oct 30 13:16:44.108786 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:44.108616    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc" event={"ID":"8daf18c1-014e-4c6b-bffb-bb25ceef69f3","Type":"ContainerStarted","Data":"2701fd856b5045dd9505a9b6fa015cbd4270105b6869939175eae5a1239fc600"}
Oct 30 13:16:44.177640 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:44.177561    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc" podStartSLOduration=2.17754079 podStartE2EDuration="2.17754079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:16:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:16:44.177274318 +0000 UTC m=+823.324541560" watchObservedRunningTime="2026-10-30 13:16:44.17754079 +0000 UTC m=+823.324808032"
Oct 30 13:16:44.209372 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:44.209219    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc"
Oct 30 13:16:44.210035 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:44.209990    5093 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 13:16:44.210596 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:44.210258    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc" podUID="8daf18c1-014e-4c6b-bffb-bb25ceef69f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 13:16:44.237658 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:44.237620    5093 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 13:16:44.237964 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:44.237916    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g" podUID="911b7cb8-3eac-4576-be8b-3fecd0fea186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 13:16:44.617556 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:44.617512    5093 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 13:16:44.617738 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:44.617586    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn" podUID="de03af69-2ff9-4add-a940-740935198076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 13:16:45.208055 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:45.208021    5093 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 13:16:45.208505 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:45.208299    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc" podUID="8daf18c1-014e-4c6b-bffb-bb25ceef69f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 13:16:45.251188 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:45.242466    5093 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 13:16:45.251188 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:45.242546    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g" podUID="911b7cb8-3eac-4576-be8b-3fecd0fea186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 13:16:45.618079 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:45.617820    5093 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 13:16:45.618079 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:45.617885    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn" podUID="de03af69-2ff9-4add-a940-740935198076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 13:16:46.210594 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:46.210539    5093 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 13:16:46.211180 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:46.210614    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc" podUID="8daf18c1-014e-4c6b-bffb-bb25ceef69f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 13:16:46.472214 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:46.471835    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g"
Oct 30 13:16:46.497320 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:46.473844    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g"
Oct 30 13:16:46.502747 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:46.502716    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g"
Oct 30 13:16:46.619572 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:46.618109    5093 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 13:16:46.619572 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:46.618208    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn" podUID="de03af69-2ff9-4add-a940-740935198076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 13:16:47.207772 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:47.207736    5093 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 13:16:47.208037 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:47.207798    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc" podUID="8daf18c1-014e-4c6b-bffb-bb25ceef69f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 13:16:47.618330 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:47.618244    5093 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 13:16:47.618330 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:47.618304    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn" podUID="de03af69-2ff9-4add-a940-740935198076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 13:16:48.207945 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:48.207908    5093 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 13:16:48.208197 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:48.207964    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc" podUID="8daf18c1-014e-4c6b-bffb-bb25ceef69f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 13:16:48.623093 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:48.623004    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn"
Oct 30 13:16:48.623531 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:48.623330    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn"
Oct 30 13:16:48.624008 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:48.623988    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn"
Oct 30 13:16:49.208407 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:49.208373    5093 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 13:16:49.208600 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:49.208429    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc" podUID="8daf18c1-014e-4c6b-bffb-bb25ceef69f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 13:16:50.207811 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:50.207770    5093 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 13:16:50.208249 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:50.207839    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc" podUID="8daf18c1-014e-4c6b-bffb-bb25ceef69f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 13:16:51.167313 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:16:51.167267    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:16:51.212977 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:51.212936    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc"
Oct 30 13:16:51.213831 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:51.213803    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc"
Oct 30 13:16:51.214400 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:16:51.214380    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc"
Oct 30 13:17:06.256054 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:06.256017    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td"]
Oct 30 13:17:21.301449 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:21.301388    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td" podUID="c64ad75a-3364-408f-b79c-46781ff2745c" containerName="main" containerID="cri-o://6f112ceed82707e9f2664d4e3d8c56e16c33dcc14eddd74ba05b16da93320781" gracePeriod=45
Oct 30 13:17:21.363362 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:17:21.363318    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:17:22.346861 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:22.346831    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td"
Oct 30 13:17:22.421326 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:22.421288    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c64ad75a-3364-408f-b79c-46781ff2745c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c64ad75a-3364-408f-b79c-46781ff2745c-model-cache\") pod \"c64ad75a-3364-408f-b79c-46781ff2745c\" (UID: \"c64ad75a-3364-408f-b79c-46781ff2745c\") "
Oct 30 13:17:22.421568 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:22.421539    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c64ad75a-3364-408f-b79c-46781ff2745c-model-cache" pod "c64ad75a-3364-408f-b79c-46781ff2745c" (UID: "c64ad75a-3364-408f-b79c-46781ff2745c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:22.421721 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:22.421591    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c64ad75a-3364-408f-b79c-46781ff2745c-kube-api-access-69v27\" (UniqueName: \"kubernetes.io/projected/c64ad75a-3364-408f-b79c-46781ff2745c-kube-api-access-69v27\") pod \"c64ad75a-3364-408f-b79c-46781ff2745c\" (UID: \"c64ad75a-3364-408f-b79c-46781ff2745c\") "
Oct 30 13:17:22.421819 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:22.421804    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c64ad75a-3364-408f-b79c-46781ff2745c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c64ad75a-3364-408f-b79c-46781ff2745c-dshm\") pod \"c64ad75a-3364-408f-b79c-46781ff2745c\" (UID: \"c64ad75a-3364-408f-b79c-46781ff2745c\") "
Oct 30 13:17:22.421913 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:22.421896    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c64ad75a-3364-408f-b79c-46781ff2745c-home\" (UniqueName: \"kubernetes.io/empty-dir/c64ad75a-3364-408f-b79c-46781ff2745c-home\") pod \"c64ad75a-3364-408f-b79c-46781ff2745c\" (UID: \"c64ad75a-3364-408f-b79c-46781ff2745c\") "
Oct 30 13:17:22.422084 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:22.422067    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c64ad75a-3364-408f-b79c-46781ff2745c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c64ad75a-3364-408f-b79c-46781ff2745c-tmp-dir\") pod \"c64ad75a-3364-408f-b79c-46781ff2745c\" (UID: \"c64ad75a-3364-408f-b79c-46781ff2745c\") "
Oct 30 13:17:22.422206 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:22.422112    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c64ad75a-3364-408f-b79c-46781ff2745c-tls-certs\" (UniqueName: \"kubernetes.io/secret/c64ad75a-3364-408f-b79c-46781ff2745c-tls-certs\") pod \"c64ad75a-3364-408f-b79c-46781ff2745c\" (UID: \"c64ad75a-3364-408f-b79c-46781ff2745c\") "
Oct 30 13:17:22.422449 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:22.422429    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c64ad75a-3364-408f-b79c-46781ff2745c-home" pod "c64ad75a-3364-408f-b79c-46781ff2745c" (UID: "c64ad75a-3364-408f-b79c-46781ff2745c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:22.422922 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:22.422894    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c64ad75a-3364-408f-b79c-46781ff2745c-tmp-dir" pod "c64ad75a-3364-408f-b79c-46781ff2745c" (UID: "c64ad75a-3364-408f-b79c-46781ff2745c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:22.423878 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:22.423855    5093 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c64ad75a-3364-408f-b79c-46781ff2745c-model-cache\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:17:22.424039 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:22.424025    5093 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c64ad75a-3364-408f-b79c-46781ff2745c-home\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:17:22.424155 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:22.424142    5093 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c64ad75a-3364-408f-b79c-46781ff2745c-tmp-dir\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:17:22.424404 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:22.424378    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c64ad75a-3364-408f-b79c-46781ff2745c-kube-api-access-69v27" pod "c64ad75a-3364-408f-b79c-46781ff2745c" (UID: "c64ad75a-3364-408f-b79c-46781ff2745c"). InnerVolumeSpecName "kube-api-access-69v27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:17:22.424828 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:22.424801    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c64ad75a-3364-408f-b79c-46781ff2745c-dshm" pod "c64ad75a-3364-408f-b79c-46781ff2745c" (UID: "c64ad75a-3364-408f-b79c-46781ff2745c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:22.426700 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:22.426679    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c64ad75a-3364-408f-b79c-46781ff2745c-tls-certs" pod "c64ad75a-3364-408f-b79c-46781ff2745c" (UID: "c64ad75a-3364-408f-b79c-46781ff2745c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:17:22.525602 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:22.525554    5093 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c64ad75a-3364-408f-b79c-46781ff2745c-dshm\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:17:22.525602 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:22.525586    5093 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c64ad75a-3364-408f-b79c-46781ff2745c-tls-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:17:22.525602 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:22.525609    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69v27\" (UniqueName: \"kubernetes.io/projected/c64ad75a-3364-408f-b79c-46781ff2745c-kube-api-access-69v27\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:17:22.536325 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:22.536284    5093 generic.go:356] "Generic (PLEG): container finished" podID="c64ad75a-3364-408f-b79c-46781ff2745c" containerID="6f112ceed82707e9f2664d4e3d8c56e16c33dcc14eddd74ba05b16da93320781" exitCode=0
Oct 30 13:17:22.536325 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:22.536324    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td"
Oct 30 13:17:22.536525 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:22.536369    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td" event={"ID":"c64ad75a-3364-408f-b79c-46781ff2745c","Type":"ContainerDied","Data":"6f112ceed82707e9f2664d4e3d8c56e16c33dcc14eddd74ba05b16da93320781"}
Oct 30 13:17:22.536525 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:22.536419    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td" event={"ID":"c64ad75a-3364-408f-b79c-46781ff2745c","Type":"ContainerDied","Data":"bf75388f12b8375a98b703d10f1f937bbce037b3aec03b24593eb4c78ed77683"}
Oct 30 13:17:22.536525 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:22.536441    5093 scope.go:122] "RemoveContainer" containerID="6f112ceed82707e9f2664d4e3d8c56e16c33dcc14eddd74ba05b16da93320781"
Oct 30 13:17:22.549094 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:22.549071    5093 scope.go:122] "RemoveContainer" containerID="6f112ceed82707e9f2664d4e3d8c56e16c33dcc14eddd74ba05b16da93320781"
Oct 30 13:17:22.549490 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:17:22.549458    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f112ceed82707e9f2664d4e3d8c56e16c33dcc14eddd74ba05b16da93320781\": container with ID starting with 6f112ceed82707e9f2664d4e3d8c56e16c33dcc14eddd74ba05b16da93320781 not found: ID does not exist" containerID="6f112ceed82707e9f2664d4e3d8c56e16c33dcc14eddd74ba05b16da93320781"
Oct 30 13:17:22.549602 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:22.549501    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f112ceed82707e9f2664d4e3d8c56e16c33dcc14eddd74ba05b16da93320781"} err="failed to get container status \"6f112ceed82707e9f2664d4e3d8c56e16c33dcc14eddd74ba05b16da93320781\": rpc error: code = NotFound desc = could not find container \"6f112ceed82707e9f2664d4e3d8c56e16c33dcc14eddd74ba05b16da93320781\": container with ID starting with 6f112ceed82707e9f2664d4e3d8c56e16c33dcc14eddd74ba05b16da93320781 not found: ID does not exist"
Oct 30 13:17:22.592898 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:22.592816    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td"]
Oct 30 13:17:22.595713 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:22.595681    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td"]
Oct 30 13:17:22.999840 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:22.999808    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c64ad75a-3364-408f-b79c-46781ff2745c" path="/var/lib/kubelet/pods/c64ad75a-3364-408f-b79c-46781ff2745c/volumes"
Oct 30 13:17:23.047093 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:23.047048    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-b-dfdb6918-kserve-56758cb765-kg9td" podUID="c64ad75a-3364-408f-b79c-46781ff2745c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:17:23.128104 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:23.128068    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b"]
Oct 30 13:17:23.129621 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:23.129592    5093 cpu_manager.go:414] "containerMap: removing container" podUID="c64ad75a-3364-408f-b79c-46781ff2745c" containerName="main"
Oct 30 13:17:23.129621 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:23.129618    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c64ad75a-3364-408f-b79c-46781ff2745c" containerName="main"
Oct 30 13:17:23.129820 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:23.129750    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="c64ad75a-3364-408f-b79c-46781ff2745c" containerName="main"
Oct 30 13:17:23.134726 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:23.134691    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b"
Oct 30 13:17:23.153969 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:23.153931    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b"]
Oct 30 13:17:23.235208 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:23.235167    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9829402b-5c78-4bef-bff1-bb1faf135ce6-istio-podinfo\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b\" (UID: \"9829402b-5c78-4bef-bff1-bb1faf135ce6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b"
Oct 30 13:17:23.235388 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:23.235234    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7txh\" (UniqueName: \"kubernetes.io/projected/9829402b-5c78-4bef-bff1-bb1faf135ce6-kube-api-access-r7txh\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b\" (UID: \"9829402b-5c78-4bef-bff1-bb1faf135ce6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b"
Oct 30 13:17:23.235484 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:23.235408    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9829402b-5c78-4bef-bff1-bb1faf135ce6-istio-data\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b\" (UID: \"9829402b-5c78-4bef-bff1-bb1faf135ce6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b"
Oct 30 13:17:23.235546 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:23.235477    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9829402b-5c78-4bef-bff1-bb1faf135ce6-workload-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b\" (UID: \"9829402b-5c78-4bef-bff1-bb1faf135ce6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b"
Oct 30 13:17:23.235599 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:23.235541    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9829402b-5c78-4bef-bff1-bb1faf135ce6-istio-envoy\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b\" (UID: \"9829402b-5c78-4bef-bff1-bb1faf135ce6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b"
Oct 30 13:17:23.235599 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:23.235588    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9829402b-5c78-4bef-bff1-bb1faf135ce6-istio-token\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b\" (UID: \"9829402b-5c78-4bef-bff1-bb1faf135ce6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b"
Oct 30 13:17:23.235689 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:23.235666    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9829402b-5c78-4bef-bff1-bb1faf135ce6-workload-certs\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b\" (UID: \"9829402b-5c78-4bef-bff1-bb1faf135ce6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b"
Oct 30 13:17:23.235728 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:23.235709    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9829402b-5c78-4bef-bff1-bb1faf135ce6-credential-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b\" (UID: \"9829402b-5c78-4bef-bff1-bb1faf135ce6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b"
Oct 30 13:17:23.235769 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:23.235727    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9829402b-5c78-4bef-bff1-bb1faf135ce6-istiod-ca-cert\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b\" (UID: \"9829402b-5c78-4bef-bff1-bb1faf135ce6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b"
Oct 30 13:17:23.337003 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:23.336892    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9829402b-5c78-4bef-bff1-bb1faf135ce6-istio-data\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b\" (UID: \"9829402b-5c78-4bef-bff1-bb1faf135ce6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b"
Oct 30 13:17:23.337003 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:23.336982    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9829402b-5c78-4bef-bff1-bb1faf135ce6-workload-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b\" (UID: \"9829402b-5c78-4bef-bff1-bb1faf135ce6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b"
Oct 30 13:17:23.337291 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:23.337035    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9829402b-5c78-4bef-bff1-bb1faf135ce6-istio-envoy\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b\" (UID: \"9829402b-5c78-4bef-bff1-bb1faf135ce6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b"
Oct 30 13:17:23.337291 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:23.337063    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9829402b-5c78-4bef-bff1-bb1faf135ce6-istio-token\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b\" (UID: \"9829402b-5c78-4bef-bff1-bb1faf135ce6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b"
Oct 30 13:17:23.337291 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:23.337113    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9829402b-5c78-4bef-bff1-bb1faf135ce6-workload-certs\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b\" (UID: \"9829402b-5c78-4bef-bff1-bb1faf135ce6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b"
Oct 30 13:17:23.337457 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:23.337424    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9829402b-5c78-4bef-bff1-bb1faf135ce6-istio-data\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b\" (UID: \"9829402b-5c78-4bef-bff1-bb1faf135ce6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b"
Oct 30 13:17:23.337457 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:23.337446    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9829402b-5c78-4bef-bff1-bb1faf135ce6-workload-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b\" (UID: \"9829402b-5c78-4bef-bff1-bb1faf135ce6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b"
Oct 30 13:17:23.337567 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:23.337486    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9829402b-5c78-4bef-bff1-bb1faf135ce6-credential-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b\" (UID: \"9829402b-5c78-4bef-bff1-bb1faf135ce6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b"
Oct 30 13:17:23.337567 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:23.337518    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9829402b-5c78-4bef-bff1-bb1faf135ce6-istiod-ca-cert\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b\" (UID: \"9829402b-5c78-4bef-bff1-bb1faf135ce6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b"
Oct 30 13:17:23.337567 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:23.337540    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9829402b-5c78-4bef-bff1-bb1faf135ce6-workload-certs\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b\" (UID: \"9829402b-5c78-4bef-bff1-bb1faf135ce6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b"
Oct 30 13:17:23.337717 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:23.337612    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9829402b-5c78-4bef-bff1-bb1faf135ce6-istio-podinfo\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b\" (UID: \"9829402b-5c78-4bef-bff1-bb1faf135ce6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b"
Oct 30 13:17:23.337717 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:23.337664    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7txh\" (UniqueName: \"kubernetes.io/projected/9829402b-5c78-4bef-bff1-bb1faf135ce6-kube-api-access-r7txh\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b\" (UID: \"9829402b-5c78-4bef-bff1-bb1faf135ce6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b"
Oct 30 13:17:23.337848 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:23.337816    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9829402b-5c78-4bef-bff1-bb1faf135ce6-credential-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b\" (UID: \"9829402b-5c78-4bef-bff1-bb1faf135ce6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b"
Oct 30 13:17:23.338256 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:23.338230    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9829402b-5c78-4bef-bff1-bb1faf135ce6-istiod-ca-cert\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b\" (UID: \"9829402b-5c78-4bef-bff1-bb1faf135ce6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b"
Oct 30 13:17:23.339811 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:23.339781    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9829402b-5c78-4bef-bff1-bb1faf135ce6-istio-envoy\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b\" (UID: \"9829402b-5c78-4bef-bff1-bb1faf135ce6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b"
Oct 30 13:17:23.340072 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:23.340055    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9829402b-5c78-4bef-bff1-bb1faf135ce6-istio-podinfo\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b\" (UID: \"9829402b-5c78-4bef-bff1-bb1faf135ce6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b"
Oct 30 13:17:23.358171 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:23.358118    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9829402b-5c78-4bef-bff1-bb1faf135ce6-istio-token\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b\" (UID: \"9829402b-5c78-4bef-bff1-bb1faf135ce6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b"
Oct 30 13:17:23.358590 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:23.358260    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7txh\" (UniqueName: \"kubernetes.io/projected/9829402b-5c78-4bef-bff1-bb1faf135ce6-kube-api-access-r7txh\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b\" (UID: \"9829402b-5c78-4bef-bff1-bb1faf135ce6\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b"
Oct 30 13:17:23.455642 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:23.455604    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b"
Oct 30 13:17:23.781563 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:23.781529    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b"]
Oct 30 13:17:23.782949 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:17:23.782925    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9829402b_5c78_4bef_bff1_bb1faf135ce6.slice/crio-2e93670e4749c3790d881722631ca74f6332bb82fc5edbe10557ca5dd4b9ca18 WatchSource:0}: Error finding container 2e93670e4749c3790d881722631ca74f6332bb82fc5edbe10557ca5dd4b9ca18: Status 404 returned error can't find the container with id 2e93670e4749c3790d881722631ca74f6332bb82fc5edbe10557ca5dd4b9ca18
Oct 30 13:17:23.796145 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:23.786413    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:17:23.796145 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:23.786595    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:17:23.796145 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:23.786685    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:17:24.583120 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:24.583076    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b" event={"ID":"9829402b-5c78-4bef-bff1-bb1faf135ce6","Type":"ContainerStarted","Data":"4713c1b37b81d8fada8a07282c052cfce3434eabcabcdbf04c7849befa5cbba1"}
Oct 30 13:17:24.584276 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:24.583901    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b" event={"ID":"9829402b-5c78-4bef-bff1-bb1faf135ce6","Type":"ContainerStarted","Data":"2e93670e4749c3790d881722631ca74f6332bb82fc5edbe10557ca5dd4b9ca18"}
Oct 30 13:17:25.457452 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:25.457396    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b"
Oct 30 13:17:25.470153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:25.457905    5093 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 13:17:25.470153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:25.457968    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b" podUID="9829402b-5c78-4bef-bff1-bb1faf135ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 13:17:26.456444 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:26.456407    5093 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 13:17:26.456864 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:26.456469    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b" podUID="9829402b-5c78-4bef-bff1-bb1faf135ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 13:17:27.456667 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:27.456623    5093 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 13:17:27.457160 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:27.456737    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b" podUID="9829402b-5c78-4bef-bff1-bb1faf135ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 13:17:28.053521 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:28.053071    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b" podStartSLOduration=5.05304938 podStartE2EDuration="5.05304938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:17:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:17:24.64652393 +0000 UTC m=+863.793791172" watchObservedRunningTime="2026-10-30 13:17:28.05304938 +0000 UTC m=+867.200316623"
Oct 30 13:17:28.058696 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:28.058660    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms"]
Oct 30 13:17:28.064337 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:28.064314    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms"
Oct 30 13:17:28.073635 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:28.073601    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms"]
Oct 30 13:17:28.158472 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:28.158424    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/616525f4-fcee-4d8e-977e-f2104a46208f-workload-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms\" (UID: \"616525f4-fcee-4d8e-977e-f2104a46208f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms"
Oct 30 13:17:28.158472 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:28.158475    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/616525f4-fcee-4d8e-977e-f2104a46208f-istio-envoy\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms\" (UID: \"616525f4-fcee-4d8e-977e-f2104a46208f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms"
Oct 30 13:17:28.158707 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:28.158536    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/616525f4-fcee-4d8e-977e-f2104a46208f-credential-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms\" (UID: \"616525f4-fcee-4d8e-977e-f2104a46208f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms"
Oct 30 13:17:28.158707 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:28.158556    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/616525f4-fcee-4d8e-977e-f2104a46208f-istiod-ca-cert\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms\" (UID: \"616525f4-fcee-4d8e-977e-f2104a46208f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms"
Oct 30 13:17:28.158707 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:28.158692    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/616525f4-fcee-4d8e-977e-f2104a46208f-workload-certs\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms\" (UID: \"616525f4-fcee-4d8e-977e-f2104a46208f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms"
Oct 30 13:17:28.158817 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:28.158804    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/616525f4-fcee-4d8e-977e-f2104a46208f-istio-token\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms\" (UID: \"616525f4-fcee-4d8e-977e-f2104a46208f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms"
Oct 30 13:17:28.158859 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:28.158843    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/616525f4-fcee-4d8e-977e-f2104a46208f-istio-data\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms\" (UID: \"616525f4-fcee-4d8e-977e-f2104a46208f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms"
Oct 30 13:17:28.158990 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:28.158972    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/616525f4-fcee-4d8e-977e-f2104a46208f-istio-podinfo\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms\" (UID: \"616525f4-fcee-4d8e-977e-f2104a46208f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms"
Oct 30 13:17:28.159036 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:28.159009    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjkxl\" (UniqueName: \"kubernetes.io/projected/616525f4-fcee-4d8e-977e-f2104a46208f-kube-api-access-rjkxl\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms\" (UID: \"616525f4-fcee-4d8e-977e-f2104a46208f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms"
Oct 30 13:17:28.259735 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:28.259696    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/616525f4-fcee-4d8e-977e-f2104a46208f-workload-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms\" (UID: \"616525f4-fcee-4d8e-977e-f2104a46208f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms"
Oct 30 13:17:28.259735 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:28.259736    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/616525f4-fcee-4d8e-977e-f2104a46208f-istio-envoy\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms\" (UID: \"616525f4-fcee-4d8e-977e-f2104a46208f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms"
Oct 30 13:17:28.260006 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:28.259762    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/616525f4-fcee-4d8e-977e-f2104a46208f-credential-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms\" (UID: \"616525f4-fcee-4d8e-977e-f2104a46208f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms"
Oct 30 13:17:28.260006 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:28.259780    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/616525f4-fcee-4d8e-977e-f2104a46208f-istiod-ca-cert\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms\" (UID: \"616525f4-fcee-4d8e-977e-f2104a46208f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms"
Oct 30 13:17:28.260006 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:28.259801    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/616525f4-fcee-4d8e-977e-f2104a46208f-workload-certs\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms\" (UID: \"616525f4-fcee-4d8e-977e-f2104a46208f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms"
Oct 30 13:17:28.260006 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:28.259837    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/616525f4-fcee-4d8e-977e-f2104a46208f-istio-token\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms\" (UID: \"616525f4-fcee-4d8e-977e-f2104a46208f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms"
Oct 30 13:17:28.260006 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:28.259883    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/616525f4-fcee-4d8e-977e-f2104a46208f-istio-data\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms\" (UID: \"616525f4-fcee-4d8e-977e-f2104a46208f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms"
Oct 30 13:17:28.260006 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:28.259973    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/616525f4-fcee-4d8e-977e-f2104a46208f-istio-podinfo\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms\" (UID: \"616525f4-fcee-4d8e-977e-f2104a46208f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms"
Oct 30 13:17:28.260351 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:28.260010    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjkxl\" (UniqueName: \"kubernetes.io/projected/616525f4-fcee-4d8e-977e-f2104a46208f-kube-api-access-rjkxl\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms\" (UID: \"616525f4-fcee-4d8e-977e-f2104a46208f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms"
Oct 30 13:17:28.260351 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:28.260244    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/616525f4-fcee-4d8e-977e-f2104a46208f-workload-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms\" (UID: \"616525f4-fcee-4d8e-977e-f2104a46208f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms"
Oct 30 13:17:28.260351 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:28.260320    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/616525f4-fcee-4d8e-977e-f2104a46208f-workload-certs\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms\" (UID: \"616525f4-fcee-4d8e-977e-f2104a46208f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms"
Oct 30 13:17:28.260540 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:28.260516    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/616525f4-fcee-4d8e-977e-f2104a46208f-istio-data\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms\" (UID: \"616525f4-fcee-4d8e-977e-f2104a46208f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms"
Oct 30 13:17:28.260635 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:28.260611    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/616525f4-fcee-4d8e-977e-f2104a46208f-credential-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms\" (UID: \"616525f4-fcee-4d8e-977e-f2104a46208f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms"
Oct 30 13:17:28.260914 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:28.260888    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/616525f4-fcee-4d8e-977e-f2104a46208f-istiod-ca-cert\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms\" (UID: \"616525f4-fcee-4d8e-977e-f2104a46208f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms"
Oct 30 13:17:28.262440 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:28.262411    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/616525f4-fcee-4d8e-977e-f2104a46208f-istio-envoy\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms\" (UID: \"616525f4-fcee-4d8e-977e-f2104a46208f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms"
Oct 30 13:17:28.263803 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:28.263767    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/616525f4-fcee-4d8e-977e-f2104a46208f-istio-podinfo\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms\" (UID: \"616525f4-fcee-4d8e-977e-f2104a46208f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms"
Oct 30 13:17:28.282359 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:28.282330    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/616525f4-fcee-4d8e-977e-f2104a46208f-istio-token\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms\" (UID: \"616525f4-fcee-4d8e-977e-f2104a46208f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms"
Oct 30 13:17:28.282492 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:28.282397    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjkxl\" (UniqueName: \"kubernetes.io/projected/616525f4-fcee-4d8e-977e-f2104a46208f-kube-api-access-rjkxl\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms\" (UID: \"616525f4-fcee-4d8e-977e-f2104a46208f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms"
Oct 30 13:17:28.382358 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:28.382252    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms"
Oct 30 13:17:28.457735 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:28.457621    5093 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 13:17:28.457735 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:28.457686    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b" podUID="9829402b-5c78-4bef-bff1-bb1faf135ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 13:17:28.756698 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:28.756580    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms"]
Oct 30 13:17:28.757519 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:28.757483    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:17:28.757638 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:28.757555    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:17:28.757638 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:28.757608    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:17:29.268156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:29.267821    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25"]
Oct 30 13:17:29.278914 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:29.278890    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25"
Oct 30 13:17:29.294711 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:29.294669    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25"]
Oct 30 13:17:29.302192 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:29.302165    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-ec41e5\"/\"default-dockercfg-hlqhx\""
Oct 30 13:17:29.302430 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:29.302412    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-ec41e5\"/\"distinct-2-w2-ec41e5-kserve-self-signed-certs\""
Oct 30 13:17:29.309434 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:29.309415    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-ec41e5\"/\"openshift-service-ca.crt\""
Oct 30 13:17:29.309613 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:29.309600    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-ec41e5\"/\"kube-root-ca.crt\""
Oct 30 13:17:29.383220 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:29.383187    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7805f89f-89d4-4754-b606-c3070f4df215-model-cache\") pod \"distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25\" (UID: \"7805f89f-89d4-4754-b606-c3070f4df215\") " pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25"
Oct 30 13:17:29.383471 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:29.383448    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7805f89f-89d4-4754-b606-c3070f4df215-tmp-dir\") pod \"distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25\" (UID: \"7805f89f-89d4-4754-b606-c3070f4df215\") " pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25"
Oct 30 13:17:29.383607 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:29.383592    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7805f89f-89d4-4754-b606-c3070f4df215-dshm\") pod \"distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25\" (UID: \"7805f89f-89d4-4754-b606-c3070f4df215\") " pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25"
Oct 30 13:17:29.383755 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:29.383738    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94pw6\" (UniqueName: \"kubernetes.io/projected/7805f89f-89d4-4754-b606-c3070f4df215-kube-api-access-94pw6\") pod \"distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25\" (UID: \"7805f89f-89d4-4754-b606-c3070f4df215\") " pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25"
Oct 30 13:17:29.383950 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:29.383935    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7805f89f-89d4-4754-b606-c3070f4df215-tls-certs\") pod \"distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25\" (UID: \"7805f89f-89d4-4754-b606-c3070f4df215\") " pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25"
Oct 30 13:17:29.384069 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:29.384056    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7805f89f-89d4-4754-b606-c3070f4df215-home\") pod \"distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25\" (UID: \"7805f89f-89d4-4754-b606-c3070f4df215\") " pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25"
Oct 30 13:17:29.459281 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:29.459244    5093 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 13:17:29.459846 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:29.459812    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b" podUID="9829402b-5c78-4bef-bff1-bb1faf135ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 13:17:29.490938 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:29.490894    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7805f89f-89d4-4754-b606-c3070f4df215-tls-certs\") pod \"distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25\" (UID: \"7805f89f-89d4-4754-b606-c3070f4df215\") " pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25"
Oct 30 13:17:29.491830 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:29.491805    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7805f89f-89d4-4754-b606-c3070f4df215-home\") pod \"distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25\" (UID: \"7805f89f-89d4-4754-b606-c3070f4df215\") " pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25"
Oct 30 13:17:29.492583 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:29.492561    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7805f89f-89d4-4754-b606-c3070f4df215-model-cache\") pod \"distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25\" (UID: \"7805f89f-89d4-4754-b606-c3070f4df215\") " pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25"
Oct 30 13:17:29.492752 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:29.492728    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7805f89f-89d4-4754-b606-c3070f4df215-tmp-dir\") pod \"distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25\" (UID: \"7805f89f-89d4-4754-b606-c3070f4df215\") " pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25"
Oct 30 13:17:29.492915 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:29.492891    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7805f89f-89d4-4754-b606-c3070f4df215-dshm\") pod \"distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25\" (UID: \"7805f89f-89d4-4754-b606-c3070f4df215\") " pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25"
Oct 30 13:17:29.493085 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:29.493054    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94pw6\" (UniqueName: \"kubernetes.io/projected/7805f89f-89d4-4754-b606-c3070f4df215-kube-api-access-94pw6\") pod \"distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25\" (UID: \"7805f89f-89d4-4754-b606-c3070f4df215\") " pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25"
Oct 30 13:17:29.494361 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:29.493522    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7805f89f-89d4-4754-b606-c3070f4df215-model-cache\") pod \"distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25\" (UID: \"7805f89f-89d4-4754-b606-c3070f4df215\") " pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25"
Oct 30 13:17:29.498224 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:29.492450    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7805f89f-89d4-4754-b606-c3070f4df215-home\") pod \"distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25\" (UID: \"7805f89f-89d4-4754-b606-c3070f4df215\") " pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25"
Oct 30 13:17:29.498375 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:29.493784    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7805f89f-89d4-4754-b606-c3070f4df215-tmp-dir\") pod \"distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25\" (UID: \"7805f89f-89d4-4754-b606-c3070f4df215\") " pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25"
Oct 30 13:17:29.512154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:29.499351    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7805f89f-89d4-4754-b606-c3070f4df215-tls-certs\") pod \"distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25\" (UID: \"7805f89f-89d4-4754-b606-c3070f4df215\") " pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25"
Oct 30 13:17:29.519361 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:29.514455    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7805f89f-89d4-4754-b606-c3070f4df215-dshm\") pod \"distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25\" (UID: \"7805f89f-89d4-4754-b606-c3070f4df215\") " pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25"
Oct 30 13:17:29.536157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:29.534077    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94pw6\" (UniqueName: \"kubernetes.io/projected/7805f89f-89d4-4754-b606-c3070f4df215-kube-api-access-94pw6\") pod \"distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25\" (UID: \"7805f89f-89d4-4754-b606-c3070f4df215\") " pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25"
Oct 30 13:17:29.609150 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:29.608944    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25"
Oct 30 13:17:29.620916 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:29.620880    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms" event={"ID":"616525f4-fcee-4d8e-977e-f2104a46208f","Type":"ContainerStarted","Data":"3048453c7208161b8ae59ceef846024fe85d7e6291ed581c66ed25e55a8efc88"}
Oct 30 13:17:29.621079 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:29.620963    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms" event={"ID":"616525f4-fcee-4d8e-977e-f2104a46208f","Type":"ContainerStarted","Data":"e901c49c268969bfaff3b2358c5239b7c0f1b893d046cf0b333932cac4c6ff7a"}
Oct 30 13:17:29.971403 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:29.971349    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms" podStartSLOduration=1.971332686 podStartE2EDuration="1.971332686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:17:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:17:29.673051235 +0000 UTC m=+868.820318499" watchObservedRunningTime="2026-10-30 13:17:29.971332686 +0000 UTC m=+869.118599926"
Oct 30 13:17:29.973957 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:29.973922    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25"]
Oct 30 13:17:29.976884 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:17:29.976857    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7805f89f_89d4_4754_b606_c3070f4df215.slice/crio-248f496f2632a99009e1d51baa789c277b32699fa36d432626c595ac5c7587f4 WatchSource:0}: Error finding container 248f496f2632a99009e1d51baa789c277b32699fa36d432626c595ac5c7587f4: Status 404 returned error can't find the container with id 248f496f2632a99009e1d51baa789c277b32699fa36d432626c595ac5c7587f4
Oct 30 13:17:30.382677 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:30.382581    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms"
Oct 30 13:17:30.384538 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:30.384512    5093 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 13:17:30.384659 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:30.384566    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms" podUID="616525f4-fcee-4d8e-977e-f2104a46208f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 13:17:30.456756 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:30.456711    5093 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 13:17:30.456939 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:30.456783    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b" podUID="9829402b-5c78-4bef-bff1-bb1faf135ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 13:17:30.654212 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:30.653752    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25" event={"ID":"7805f89f-89d4-4754-b606-c3070f4df215","Type":"ContainerStarted","Data":"3275d31c19ab423200a8e6cc380046b4de705485c42a5c0fb111eded99e1d24a"}
Oct 30 13:17:30.654212 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:30.653804    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25" event={"ID":"7805f89f-89d4-4754-b606-c3070f4df215","Type":"ContainerStarted","Data":"248f496f2632a99009e1d51baa789c277b32699fa36d432626c595ac5c7587f4"}
Oct 30 13:17:30.707532 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:30.707469    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25" podStartSLOduration=1.707449174 podStartE2EDuration="1.707449174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:17:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:17:30.706580319 +0000 UTC m=+869.853847560" watchObservedRunningTime="2026-10-30 13:17:30.707449174 +0000 UTC m=+869.854716416"
Oct 30 13:17:31.386151 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:31.382884    5093 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 13:17:31.386151 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:31.382951    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms" podUID="616525f4-fcee-4d8e-977e-f2104a46208f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 13:17:31.548155 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:31.546618    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b"
Oct 30 13:17:31.548155 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:31.547728    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b"
Oct 30 13:17:31.561801 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:31.561671    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b"
Oct 30 13:17:31.609453 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:31.609415    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25"
Oct 30 13:17:32.383351 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:32.383310    5093 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 13:17:32.383852 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:32.383370    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms" podUID="616525f4-fcee-4d8e-977e-f2104a46208f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 13:17:33.383573 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:33.383533    5093 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 13:17:33.384015 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:33.383598    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms" podUID="616525f4-fcee-4d8e-977e-f2104a46208f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 13:17:34.383501 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:34.383460    5093 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 13:17:34.383693 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:34.383534    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms" podUID="616525f4-fcee-4d8e-977e-f2104a46208f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 13:17:35.383434 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:35.383398    5093 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 13:17:35.383611 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:35.383456    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms" podUID="616525f4-fcee-4d8e-977e-f2104a46208f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 13:17:36.387224 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:36.387191    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms"
Oct 30 13:17:36.387625 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:36.387592    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms"
Oct 30 13:17:36.388333 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:36.388312    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms"
Oct 30 13:17:39.609617 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:39.609581    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25"
Oct 30 13:17:39.680285 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:39.680225    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx" podUID="ba156eeb-9955-4f9a-857b-2c4d502a5fcc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:17:39.729507 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:39.729461    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv" podUID="d6a44981-104f-4d3e-9937-d184a6ddc482" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:17:39.797156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:39.793575    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt" podUID="4ae52c69-0c30-438f-9ac0-c87b6de16061" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 30 13:17:39.797156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:39.795254    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw" podUID="ad37954e-1350-4f75-a63f-a0c0f29ae416" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 13:17:40.142486 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:40.142453    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25"
Oct 30 13:17:40.681280 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:40.681228    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx" podUID="ba156eeb-9955-4f9a-857b-2c4d502a5fcc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 30 13:17:40.710155 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:40.682675    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx"
Oct 30 13:17:40.800274 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:40.799912    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx"
Oct 30 13:17:40.855838 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:40.823798    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt" podUID="4ae52c69-0c30-438f-9ac0-c87b6de16061" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 30 13:17:40.859975 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:40.859944    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt"
Oct 30 13:17:40.892691 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:40.890467    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25"
Oct 30 13:17:40.896652 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:40.896516    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt"
Oct 30 13:17:46.977473 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:46.977441    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb"]
Oct 30 13:17:46.981605 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:46.981581    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb"
Oct 30 13:17:47.003248 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:47.003220    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb"]
Oct 30 13:17:47.008469 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:47.008439    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9bf8ad\"/\"distinct-w4-9bf8ad-kserve-self-signed-certs\""
Oct 30 13:17:47.075786 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:47.075754    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4b4e00e-f58d-4a4e-b7db-19f724360db1-model-cache\") pod \"distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb\" (UID: \"c4b4e00e-f58d-4a4e-b7db-19f724360db1\") " pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb"
Oct 30 13:17:47.075985 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:47.075800    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4b4e00e-f58d-4a4e-b7db-19f724360db1-tls-certs\") pod \"distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb\" (UID: \"c4b4e00e-f58d-4a4e-b7db-19f724360db1\") " pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb"
Oct 30 13:17:47.075985 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:47.075825    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4b4e00e-f58d-4a4e-b7db-19f724360db1-home\") pod \"distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb\" (UID: \"c4b4e00e-f58d-4a4e-b7db-19f724360db1\") " pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb"
Oct 30 13:17:47.076097 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:47.076003    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcgxf\" (UniqueName: \"kubernetes.io/projected/c4b4e00e-f58d-4a4e-b7db-19f724360db1-kube-api-access-tcgxf\") pod \"distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb\" (UID: \"c4b4e00e-f58d-4a4e-b7db-19f724360db1\") " pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb"
Oct 30 13:17:47.076097 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:47.076068    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4b4e00e-f58d-4a4e-b7db-19f724360db1-tmp-dir\") pod \"distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb\" (UID: \"c4b4e00e-f58d-4a4e-b7db-19f724360db1\") " pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb"
Oct 30 13:17:47.076097 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:47.076091    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4b4e00e-f58d-4a4e-b7db-19f724360db1-dshm\") pod \"distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb\" (UID: \"c4b4e00e-f58d-4a4e-b7db-19f724360db1\") " pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb"
Oct 30 13:17:47.177711 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:47.177674    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcgxf\" (UniqueName: \"kubernetes.io/projected/c4b4e00e-f58d-4a4e-b7db-19f724360db1-kube-api-access-tcgxf\") pod \"distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb\" (UID: \"c4b4e00e-f58d-4a4e-b7db-19f724360db1\") " pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb"
Oct 30 13:17:47.177935 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:47.177774    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4b4e00e-f58d-4a4e-b7db-19f724360db1-tmp-dir\") pod \"distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb\" (UID: \"c4b4e00e-f58d-4a4e-b7db-19f724360db1\") " pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb"
Oct 30 13:17:47.177935 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:47.177805    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4b4e00e-f58d-4a4e-b7db-19f724360db1-dshm\") pod \"distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb\" (UID: \"c4b4e00e-f58d-4a4e-b7db-19f724360db1\") " pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb"
Oct 30 13:17:47.177935 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:47.177902    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4b4e00e-f58d-4a4e-b7db-19f724360db1-model-cache\") pod \"distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb\" (UID: \"c4b4e00e-f58d-4a4e-b7db-19f724360db1\") " pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb"
Oct 30 13:17:47.178122 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:47.177941    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4b4e00e-f58d-4a4e-b7db-19f724360db1-tls-certs\") pod \"distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb\" (UID: \"c4b4e00e-f58d-4a4e-b7db-19f724360db1\") " pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb"
Oct 30 13:17:47.178122 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:47.177974    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4b4e00e-f58d-4a4e-b7db-19f724360db1-home\") pod \"distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb\" (UID: \"c4b4e00e-f58d-4a4e-b7db-19f724360db1\") " pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb"
Oct 30 13:17:47.178295 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:47.178267    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4b4e00e-f58d-4a4e-b7db-19f724360db1-tmp-dir\") pod \"distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb\" (UID: \"c4b4e00e-f58d-4a4e-b7db-19f724360db1\") " pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb"
Oct 30 13:17:47.178437 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:47.178343    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4b4e00e-f58d-4a4e-b7db-19f724360db1-model-cache\") pod \"distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb\" (UID: \"c4b4e00e-f58d-4a4e-b7db-19f724360db1\") " pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb"
Oct 30 13:17:47.178437 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:47.178389    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4b4e00e-f58d-4a4e-b7db-19f724360db1-home\") pod \"distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb\" (UID: \"c4b4e00e-f58d-4a4e-b7db-19f724360db1\") " pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb"
Oct 30 13:17:47.180190 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:47.180166    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4b4e00e-f58d-4a4e-b7db-19f724360db1-dshm\") pod \"distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb\" (UID: \"c4b4e00e-f58d-4a4e-b7db-19f724360db1\") " pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb"
Oct 30 13:17:47.180500 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:47.180483    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4b4e00e-f58d-4a4e-b7db-19f724360db1-tls-certs\") pod \"distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb\" (UID: \"c4b4e00e-f58d-4a4e-b7db-19f724360db1\") " pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb"
Oct 30 13:17:47.197952 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:47.197926    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcgxf\" (UniqueName: \"kubernetes.io/projected/c4b4e00e-f58d-4a4e-b7db-19f724360db1-kube-api-access-tcgxf\") pod \"distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb\" (UID: \"c4b4e00e-f58d-4a4e-b7db-19f724360db1\") " pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb"
Oct 30 13:17:47.298842 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:47.298751    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb"
Oct 30 13:17:47.604207 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:47.604179    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb"]
Oct 30 13:17:47.606068 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:17:47.606031    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4b4e00e_f58d_4a4e_b7db_19f724360db1.slice/crio-abcd0c618bfc497c28340c151fd4f319158ff79ac00917bafb06d5081257d4fd WatchSource:0}: Error finding container abcd0c618bfc497c28340c151fd4f319158ff79ac00917bafb06d5081257d4fd: Status 404 returned error can't find the container with id abcd0c618bfc497c28340c151fd4f319158ff79ac00917bafb06d5081257d4fd
Oct 30 13:17:47.925797 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:47.925759    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb" event={"ID":"c4b4e00e-f58d-4a4e-b7db-19f724360db1","Type":"ContainerStarted","Data":"b2a5c79681857ff27077fb57511fd57f01046ba1192945042b9122a373391b3e"}
Oct 30 13:17:47.925984 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:47.925806    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb" event={"ID":"c4b4e00e-f58d-4a4e-b7db-19f724360db1","Type":"ContainerStarted","Data":"abcd0c618bfc497c28340c151fd4f319158ff79ac00917bafb06d5081257d4fd"}
Oct 30 13:17:47.966781 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:47.966722    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb" podStartSLOduration=1.96670618 podStartE2EDuration="1.96670618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:17:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:17:47.966324628 +0000 UTC m=+887.113591871" watchObservedRunningTime="2026-10-30 13:17:47.96670618 +0000 UTC m=+887.113973421"
Oct 30 13:17:48.299738 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:48.299642    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb"
Oct 30 13:17:51.167673 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:17:51.167628    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:17:53.540192 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:53.540151    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr"]
Oct 30 13:17:53.544446 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:53.544419    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr"
Oct 30 13:17:53.563866 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:53.563833    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr"]
Oct 30 13:17:53.685989 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:53.685951    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b646832b-e0d1-491d-bb54-86546c08081e-istio-envoy\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr\" (UID: \"b646832b-e0d1-491d-bb54-86546c08081e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr"
Oct 30 13:17:53.686191 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:53.686012    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b646832b-e0d1-491d-bb54-86546c08081e-workload-certs\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr\" (UID: \"b646832b-e0d1-491d-bb54-86546c08081e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr"
Oct 30 13:17:53.686191 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:53.686144    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b646832b-e0d1-491d-bb54-86546c08081e-istio-data\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr\" (UID: \"b646832b-e0d1-491d-bb54-86546c08081e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr"
Oct 30 13:17:53.686301 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:53.686201    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b646832b-e0d1-491d-bb54-86546c08081e-credential-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr\" (UID: \"b646832b-e0d1-491d-bb54-86546c08081e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr"
Oct 30 13:17:53.686361 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:53.686323    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b646832b-e0d1-491d-bb54-86546c08081e-istio-podinfo\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr\" (UID: \"b646832b-e0d1-491d-bb54-86546c08081e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr"
Oct 30 13:17:53.686419 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:53.686365    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b646832b-e0d1-491d-bb54-86546c08081e-istiod-ca-cert\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr\" (UID: \"b646832b-e0d1-491d-bb54-86546c08081e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr"
Oct 30 13:17:53.686487 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:53.686470    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b646832b-e0d1-491d-bb54-86546c08081e-workload-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr\" (UID: \"b646832b-e0d1-491d-bb54-86546c08081e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr"
Oct 30 13:17:53.686596 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:53.686581    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b646832b-e0d1-491d-bb54-86546c08081e-istio-token\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr\" (UID: \"b646832b-e0d1-491d-bb54-86546c08081e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr"
Oct 30 13:17:53.686657 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:53.686633    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kkrp\" (UniqueName: \"kubernetes.io/projected/b646832b-e0d1-491d-bb54-86546c08081e-kube-api-access-5kkrp\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr\" (UID: \"b646832b-e0d1-491d-bb54-86546c08081e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr"
Oct 30 13:17:53.788436 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:53.788398    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b646832b-e0d1-491d-bb54-86546c08081e-istio-token\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr\" (UID: \"b646832b-e0d1-491d-bb54-86546c08081e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr"
Oct 30 13:17:53.788646 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:53.788474    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kkrp\" (UniqueName: \"kubernetes.io/projected/b646832b-e0d1-491d-bb54-86546c08081e-kube-api-access-5kkrp\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr\" (UID: \"b646832b-e0d1-491d-bb54-86546c08081e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr"
Oct 30 13:17:53.788646 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:53.788559    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b646832b-e0d1-491d-bb54-86546c08081e-istio-envoy\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr\" (UID: \"b646832b-e0d1-491d-bb54-86546c08081e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr"
Oct 30 13:17:53.788646 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:53.788624    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b646832b-e0d1-491d-bb54-86546c08081e-workload-certs\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr\" (UID: \"b646832b-e0d1-491d-bb54-86546c08081e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr"
Oct 30 13:17:53.788849 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:53.788668    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b646832b-e0d1-491d-bb54-86546c08081e-istio-data\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr\" (UID: \"b646832b-e0d1-491d-bb54-86546c08081e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr"
Oct 30 13:17:53.788849 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:53.788715    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b646832b-e0d1-491d-bb54-86546c08081e-credential-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr\" (UID: \"b646832b-e0d1-491d-bb54-86546c08081e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr"
Oct 30 13:17:53.788849 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:53.788762    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b646832b-e0d1-491d-bb54-86546c08081e-istio-podinfo\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr\" (UID: \"b646832b-e0d1-491d-bb54-86546c08081e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr"
Oct 30 13:17:53.788849 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:53.788789    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b646832b-e0d1-491d-bb54-86546c08081e-istiod-ca-cert\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr\" (UID: \"b646832b-e0d1-491d-bb54-86546c08081e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr"
Oct 30 13:17:53.789051 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:53.788851    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b646832b-e0d1-491d-bb54-86546c08081e-workload-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr\" (UID: \"b646832b-e0d1-491d-bb54-86546c08081e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr"
Oct 30 13:17:53.789248 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:53.789202    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b646832b-e0d1-491d-bb54-86546c08081e-workload-certs\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr\" (UID: \"b646832b-e0d1-491d-bb54-86546c08081e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr"
Oct 30 13:17:53.789248 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:53.789210    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b646832b-e0d1-491d-bb54-86546c08081e-workload-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr\" (UID: \"b646832b-e0d1-491d-bb54-86546c08081e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr"
Oct 30 13:17:53.789447 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:53.789295    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b646832b-e0d1-491d-bb54-86546c08081e-credential-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr\" (UID: \"b646832b-e0d1-491d-bb54-86546c08081e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr"
Oct 30 13:17:53.789645 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:53.789616    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b646832b-e0d1-491d-bb54-86546c08081e-istio-data\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr\" (UID: \"b646832b-e0d1-491d-bb54-86546c08081e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr"
Oct 30 13:17:53.789908 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:53.789886    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b646832b-e0d1-491d-bb54-86546c08081e-istiod-ca-cert\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr\" (UID: \"b646832b-e0d1-491d-bb54-86546c08081e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr"
Oct 30 13:17:53.791295 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:53.791232    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b646832b-e0d1-491d-bb54-86546c08081e-istio-envoy\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr\" (UID: \"b646832b-e0d1-491d-bb54-86546c08081e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr"
Oct 30 13:17:53.791795 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:53.791781    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b646832b-e0d1-491d-bb54-86546c08081e-istio-podinfo\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr\" (UID: \"b646832b-e0d1-491d-bb54-86546c08081e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr"
Oct 30 13:17:53.808782 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:53.808746    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b646832b-e0d1-491d-bb54-86546c08081e-istio-token\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr\" (UID: \"b646832b-e0d1-491d-bb54-86546c08081e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr"
Oct 30 13:17:53.808921 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:53.808899    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kkrp\" (UniqueName: \"kubernetes.io/projected/b646832b-e0d1-491d-bb54-86546c08081e-kube-api-access-5kkrp\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr\" (UID: \"b646832b-e0d1-491d-bb54-86546c08081e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr"
Oct 30 13:17:53.861693 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:53.861653    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr"
Oct 30 13:17:54.180828 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:54.180794    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr"]
Oct 30 13:17:54.183072 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:17:54.183045    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb646832b_e0d1_491d_bb54_86546c08081e.slice/crio-6b98599e6077a0563feaa71263c32c04ebc2c9de20794c3a79005d1f2890d1fb WatchSource:0}: Error finding container 6b98599e6077a0563feaa71263c32c04ebc2c9de20794c3a79005d1f2890d1fb: Status 404 returned error can't find the container with id 6b98599e6077a0563feaa71263c32c04ebc2c9de20794c3a79005d1f2890d1fb
Oct 30 13:17:54.185589 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:54.185557    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:17:54.185664 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:54.185626    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:17:54.185664 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:54.185654    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:17:55.000788 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:55.000749    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr" event={"ID":"b646832b-e0d1-491d-bb54-86546c08081e","Type":"ContainerStarted","Data":"6fdbb6b17e3c20959727a1be710c4b798ed9302f9edff38cf4694ac31edfa962"}
Oct 30 13:17:55.001214 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:55.000813    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr" event={"ID":"b646832b-e0d1-491d-bb54-86546c08081e","Type":"ContainerStarted","Data":"6b98599e6077a0563feaa71263c32c04ebc2c9de20794c3a79005d1f2890d1fb"}
Oct 30 13:17:55.067974 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:55.067898    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr" podStartSLOduration=2.067876228 podStartE2EDuration="2.067876228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:17:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:17:55.065337513 +0000 UTC m=+894.212604755" watchObservedRunningTime="2026-10-30 13:17:55.067876228 +0000 UTC m=+894.215143471"
Oct 30 13:17:55.866693 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:55.865633    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr"
Oct 30 13:17:55.866693 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:55.865813    5093 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 13:17:55.866693 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:55.865875    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr" podUID="b646832b-e0d1-491d-bb54-86546c08081e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 13:17:56.902156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:56.898275    5093 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 13:17:56.902156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:56.898363    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr" podUID="b646832b-e0d1-491d-bb54-86546c08081e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 13:17:57.349157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:57.346239    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb"
Oct 30 13:17:57.683301 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:57.683246    5093 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg" podUID="ea079ac1-f502-4b5f-b16a-c91e2de98de8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:17:57.715271 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:57.715222    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg" podUID="ea079ac1-f502-4b5f-b16a-c91e2de98de8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:17:57.722167 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:57.721214    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv" podUID="d6a44981-104f-4d3e-9937-d184a6ddc482" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:17:57.800170 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:57.796769    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw" podUID="ad37954e-1350-4f75-a63f-a0c0f29ae416" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:17:57.805256 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:57.805207    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx" podUID="ba156eeb-9955-4f9a-857b-2c4d502a5fcc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:17:57.864700 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:57.864661    5093 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 13:17:57.864973 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:57.864938    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr" podUID="b646832b-e0d1-491d-bb54-86546c08081e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 13:17:57.917282 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:57.917230    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt" podUID="4ae52c69-0c30-438f-9ac0-c87b6de16061" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:17:58.152648 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:58.152534    5093 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx" podUID="ba156eeb-9955-4f9a-857b-2c4d502a5fcc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:17:58.152947 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:58.152924    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25" podUID="7805f89f-89d4-4754-b606-c3070f4df215" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:17:58.172246 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:58.167158    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb"
Oct 30 13:17:58.887150 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:58.886404    5093 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 13:17:58.887150 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:58.886488    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr" podUID="b646832b-e0d1-491d-bb54-86546c08081e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 13:17:59.031152 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:59.006195    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt" podUID="4ae52c69-0c30-438f-9ac0-c87b6de16061" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 30 13:17:59.165208 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:59.157045    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25" podUID="7805f89f-89d4-4754-b606-c3070f4df215" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 13:17:59.345156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:59.339750    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb" podUID="c4b4e00e-f58d-4a4e-b7db-19f724360db1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:17:59.373155 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:59.369252    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt"
Oct 30 13:17:59.373155 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:59.369320    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25"
Oct 30 13:17:59.725162 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:59.724300    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv" podUID="d6a44981-104f-4d3e-9937-d184a6ddc482" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 13:17:59.725376 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:59.725246    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg" podUID="ea079ac1-f502-4b5f-b16a-c91e2de98de8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:17:59.901153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:59.898421    5093 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 13:17:59.901153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:59.898503    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr" podUID="b646832b-e0d1-491d-bb54-86546c08081e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 13:17:59.921785 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:59.920252    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx" podUID="ba156eeb-9955-4f9a-857b-2c4d502a5fcc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 30 13:17:59.931917 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:17:59.930236    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw" podUID="ad37954e-1350-4f75-a63f-a0c0f29ae416" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:00.013242 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:00.010489    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt" podUID="4ae52c69-0c30-438f-9ac0-c87b6de16061" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 30 13:18:00.198591 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:00.198548    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25" podUID="7805f89f-89d4-4754-b606-c3070f4df215" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 13:18:00.307402 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:00.307072    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25"
Oct 30 13:18:00.341429 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:00.341383    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb" podUID="c4b4e00e-f58d-4a4e-b7db-19f724360db1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:00.391601 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:00.391565    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt"
Oct 30 13:18:00.402404 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:00.402371    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb"
Oct 30 13:18:00.862656 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:00.862614    5093 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 13:18:00.862874 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:00.862687    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr" podUID="b646832b-e0d1-491d-bb54-86546c08081e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 13:18:01.174746 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:01.174715    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jdj4_f93ee017-ab79-447c-90e0-5b85b71dba24/console-operator/1.log"
Oct 30 13:18:01.175481 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:01.175459    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jdj4_f93ee017-ab79-447c-90e0-5b85b71dba24/console-operator/1.log"
Oct 30 13:18:01.183107 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:01.183076    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tnm_3a675ac7-fbe1-449b-accf-bb0456ed0bd3/ovn-acl-logging/0.log"
Oct 30 13:18:01.184039 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:01.184014    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tnm_3a675ac7-fbe1-449b-accf-bb0456ed0bd3/ovn-acl-logging/0.log"
Oct 30 13:18:01.862582 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:01.862540    5093 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 13:18:01.863015 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:01.862611    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr" podUID="b646832b-e0d1-491d-bb54-86546c08081e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 13:18:02.862857 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:02.862810    5093 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 13:18:02.863399 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:02.862883    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr" podUID="b646832b-e0d1-491d-bb54-86546c08081e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 13:18:03.867752 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:03.867717    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr"
Oct 30 13:18:03.868358 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:03.868235    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr"
Oct 30 13:18:03.868933 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:03.868904    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr"
Oct 30 13:18:05.646694 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:05.646657    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn"]
Oct 30 13:18:05.653692 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:05.653656    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn"
Oct 30 13:18:05.678036 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:05.677996    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn"]
Oct 30 13:18:05.818767 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:05.818723    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9980a626-91cc-492f-8918-1970311586e6-workload-certs\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn\" (UID: \"9980a626-91cc-492f-8918-1970311586e6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn"
Oct 30 13:18:05.818968 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:05.818856    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc6s2\" (UniqueName: \"kubernetes.io/projected/9980a626-91cc-492f-8918-1970311586e6-kube-api-access-xc6s2\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn\" (UID: \"9980a626-91cc-492f-8918-1970311586e6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn"
Oct 30 13:18:05.819034 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:05.818976    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9980a626-91cc-492f-8918-1970311586e6-istio-envoy\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn\" (UID: \"9980a626-91cc-492f-8918-1970311586e6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn"
Oct 30 13:18:05.819115 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:05.819097    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9980a626-91cc-492f-8918-1970311586e6-istiod-ca-cert\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn\" (UID: \"9980a626-91cc-492f-8918-1970311586e6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn"
Oct 30 13:18:05.819215 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:05.819124    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9980a626-91cc-492f-8918-1970311586e6-istio-data\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn\" (UID: \"9980a626-91cc-492f-8918-1970311586e6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn"
Oct 30 13:18:05.819281 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:05.819265    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9980a626-91cc-492f-8918-1970311586e6-istio-token\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn\" (UID: \"9980a626-91cc-492f-8918-1970311586e6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn"
Oct 30 13:18:05.819321 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:05.819300    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9980a626-91cc-492f-8918-1970311586e6-workload-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn\" (UID: \"9980a626-91cc-492f-8918-1970311586e6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn"
Oct 30 13:18:05.819378 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:05.819323    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9980a626-91cc-492f-8918-1970311586e6-credential-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn\" (UID: \"9980a626-91cc-492f-8918-1970311586e6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn"
Oct 30 13:18:05.819378 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:05.819340    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9980a626-91cc-492f-8918-1970311586e6-istio-podinfo\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn\" (UID: \"9980a626-91cc-492f-8918-1970311586e6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn"
Oct 30 13:18:05.921001 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:05.920917    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9980a626-91cc-492f-8918-1970311586e6-istiod-ca-cert\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn\" (UID: \"9980a626-91cc-492f-8918-1970311586e6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn"
Oct 30 13:18:05.921001 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:05.920953    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9980a626-91cc-492f-8918-1970311586e6-istio-data\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn\" (UID: \"9980a626-91cc-492f-8918-1970311586e6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn"
Oct 30 13:18:05.921293 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:05.921014    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9980a626-91cc-492f-8918-1970311586e6-istio-token\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn\" (UID: \"9980a626-91cc-492f-8918-1970311586e6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn"
Oct 30 13:18:05.921293 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:05.921039    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9980a626-91cc-492f-8918-1970311586e6-workload-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn\" (UID: \"9980a626-91cc-492f-8918-1970311586e6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn"
Oct 30 13:18:05.921293 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:05.921054    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9980a626-91cc-492f-8918-1970311586e6-credential-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn\" (UID: \"9980a626-91cc-492f-8918-1970311586e6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn"
Oct 30 13:18:05.921293 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:05.921079    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9980a626-91cc-492f-8918-1970311586e6-istio-podinfo\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn\" (UID: \"9980a626-91cc-492f-8918-1970311586e6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn"
Oct 30 13:18:05.921293 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:05.921196    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9980a626-91cc-492f-8918-1970311586e6-workload-certs\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn\" (UID: \"9980a626-91cc-492f-8918-1970311586e6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn"
Oct 30 13:18:05.921293 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:05.921247    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc6s2\" (UniqueName: \"kubernetes.io/projected/9980a626-91cc-492f-8918-1970311586e6-kube-api-access-xc6s2\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn\" (UID: \"9980a626-91cc-492f-8918-1970311586e6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn"
Oct 30 13:18:05.921610 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:05.921333    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9980a626-91cc-492f-8918-1970311586e6-istio-envoy\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn\" (UID: \"9980a626-91cc-492f-8918-1970311586e6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn"
Oct 30 13:18:05.921610 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:05.921501    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9980a626-91cc-492f-8918-1970311586e6-istio-data\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn\" (UID: \"9980a626-91cc-492f-8918-1970311586e6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn"
Oct 30 13:18:05.921610 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:05.921595    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9980a626-91cc-492f-8918-1970311586e6-credential-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn\" (UID: \"9980a626-91cc-492f-8918-1970311586e6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn"
Oct 30 13:18:05.921732 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:05.921702    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9980a626-91cc-492f-8918-1970311586e6-workload-certs\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn\" (UID: \"9980a626-91cc-492f-8918-1970311586e6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn"
Oct 30 13:18:05.921732 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:05.921715    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9980a626-91cc-492f-8918-1970311586e6-workload-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn\" (UID: \"9980a626-91cc-492f-8918-1970311586e6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn"
Oct 30 13:18:05.921879 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:05.921859    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9980a626-91cc-492f-8918-1970311586e6-istiod-ca-cert\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn\" (UID: \"9980a626-91cc-492f-8918-1970311586e6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn"
Oct 30 13:18:05.923640 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:05.923622    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9980a626-91cc-492f-8918-1970311586e6-istio-envoy\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn\" (UID: \"9980a626-91cc-492f-8918-1970311586e6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn"
Oct 30 13:18:05.923899 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:05.923875    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9980a626-91cc-492f-8918-1970311586e6-istio-podinfo\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn\" (UID: \"9980a626-91cc-492f-8918-1970311586e6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn"
Oct 30 13:18:05.941198 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:05.941168    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9980a626-91cc-492f-8918-1970311586e6-istio-token\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn\" (UID: \"9980a626-91cc-492f-8918-1970311586e6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn"
Oct 30 13:18:05.941342 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:05.941284    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc6s2\" (UniqueName: \"kubernetes.io/projected/9980a626-91cc-492f-8918-1970311586e6-kube-api-access-xc6s2\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn\" (UID: \"9980a626-91cc-492f-8918-1970311586e6\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn"
Oct 30 13:18:05.968537 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:05.968500    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn"
Oct 30 13:18:06.290618 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:06.290582    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn"]
Oct 30 13:18:06.291829 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:18:06.291801    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9980a626_91cc_492f_8918_1970311586e6.slice/crio-e29378a23617c2c525a4aec3b6dc9272395ededf5a17025b209fb0d7bbb37f20 WatchSource:0}: Error finding container e29378a23617c2c525a4aec3b6dc9272395ededf5a17025b209fb0d7bbb37f20: Status 404 returned error can't find the container with id e29378a23617c2c525a4aec3b6dc9272395ededf5a17025b209fb0d7bbb37f20
Oct 30 13:18:06.294827 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:06.294791    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:18:06.294931 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:06.294878    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:18:06.295001 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:06.294927    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:18:06.597052 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:06.596959    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn" event={"ID":"9980a626-91cc-492f-8918-1970311586e6","Type":"ContainerStarted","Data":"8cc739053c7972db5bf9dcc26c702a7eb70b3ef68338b1e2490bbe7773e73853"}
Oct 30 13:18:06.597052 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:06.597023    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn" event={"ID":"9980a626-91cc-492f-8918-1970311586e6","Type":"ContainerStarted","Data":"e29378a23617c2c525a4aec3b6dc9272395ededf5a17025b209fb0d7bbb37f20"}
Oct 30 13:18:06.645503 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:06.645438    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn" podStartSLOduration=1.6454240329999998 podStartE2EDuration="1.645424033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:06.642266952 +0000 UTC m=+905.789534196" watchObservedRunningTime="2026-10-30 13:18:06.645424033 +0000 UTC m=+905.792691273"
Oct 30 13:18:06.968867 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:06.968826    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn"
Oct 30 13:18:07.970636 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:07.970593    5093 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 13:18:07.971042 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:07.970653    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn" podUID="9980a626-91cc-492f-8918-1970311586e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 13:18:08.969693 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:08.969644    5093 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 13:18:08.969911 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:08.969720    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn" podUID="9980a626-91cc-492f-8918-1970311586e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 13:18:09.969840 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:09.969803    5093 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 13:18:09.970283 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:09.969872    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn" podUID="9980a626-91cc-492f-8918-1970311586e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 13:18:10.969522 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:10.969485    5093 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 13:18:10.969716 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:10.969552    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn" podUID="9980a626-91cc-492f-8918-1970311586e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 13:18:11.969551 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:11.969509    5093 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 13:18:11.969960 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:11.969570    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn" podUID="9980a626-91cc-492f-8918-1970311586e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 13:18:13.266719 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:13.266675    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls"]
Oct 30 13:18:13.271188 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:13.271162    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls"
Oct 30 13:18:13.295110 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:13.295055    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls"]
Oct 30 13:18:13.302768 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:13.302679    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn"
Oct 30 13:18:13.308786 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:13.308756    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn"
Oct 30 13:18:13.309812 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:13.309782    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn"
Oct 30 13:18:13.375934 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:13.375896    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-workload-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls\" (UID: \"c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls"
Oct 30 13:18:13.376177 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:13.375938    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jf5k\" (UniqueName: \"kubernetes.io/projected/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-kube-api-access-5jf5k\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls\" (UID: \"c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls"
Oct 30 13:18:13.376888 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:13.376857    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-istio-data\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls\" (UID: \"c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls"
Oct 30 13:18:13.377018 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:13.376920    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-credential-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls\" (UID: \"c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls"
Oct 30 13:18:13.377018 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:13.376975    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-istiod-ca-cert\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls\" (UID: \"c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls"
Oct 30 13:18:13.377193 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:13.377122    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-istio-envoy\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls\" (UID: \"c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls"
Oct 30 13:18:13.377407 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:13.377343    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-istio-podinfo\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls\" (UID: \"c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls"
Oct 30 13:18:13.377476 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:13.377413    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-workload-certs\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls\" (UID: \"c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls"
Oct 30 13:18:13.377476 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:13.377461    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-istio-token\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls\" (UID: \"c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls"
Oct 30 13:18:13.478506 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:13.478467    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-istio-podinfo\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls\" (UID: \"c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls"
Oct 30 13:18:13.478506 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:13.478523    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-workload-certs\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls\" (UID: \"c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls"
Oct 30 13:18:13.478767 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:13.478545    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-istio-token\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls\" (UID: \"c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls"
Oct 30 13:18:13.478767 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:13.478605    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-workload-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls\" (UID: \"c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls"
Oct 30 13:18:13.478767 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:13.478621    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jf5k\" (UniqueName: \"kubernetes.io/projected/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-kube-api-access-5jf5k\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls\" (UID: \"c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls"
Oct 30 13:18:13.478767 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:13.478674    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-istio-data\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls\" (UID: \"c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls"
Oct 30 13:18:13.478767 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:13.478707    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-credential-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls\" (UID: \"c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls"
Oct 30 13:18:13.478767 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:13.478754    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-istiod-ca-cert\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls\" (UID: \"c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls"
Oct 30 13:18:13.479178 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:13.478814    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-istio-envoy\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls\" (UID: \"c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls"
Oct 30 13:18:13.479178 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:13.478992    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-workload-certs\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls\" (UID: \"c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls"
Oct 30 13:18:13.479299 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:13.479205    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-istio-data\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls\" (UID: \"c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls"
Oct 30 13:18:13.479586 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:13.479554    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-credential-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls\" (UID: \"c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls"
Oct 30 13:18:13.479722 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:13.479613    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-istiod-ca-cert\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls\" (UID: \"c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls"
Oct 30 13:18:13.479722 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:13.479666    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-workload-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls\" (UID: \"c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls"
Oct 30 13:18:13.481028 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:13.481005    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-istio-envoy\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls\" (UID: \"c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls"
Oct 30 13:18:13.481170 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:13.481072    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-istio-podinfo\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls\" (UID: \"c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls"
Oct 30 13:18:13.499361 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:13.499116    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-istio-token\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls\" (UID: \"c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls"
Oct 30 13:18:13.499361 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:13.499280    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jf5k\" (UniqueName: \"kubernetes.io/projected/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-kube-api-access-5jf5k\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls\" (UID: \"c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls"
Oct 30 13:18:13.609050 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:13.608951    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls"
Oct 30 13:18:13.932645 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:13.932612    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls"]
Oct 30 13:18:13.933524 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:18:13.933497    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc49a3d68_55ef_4c8a_8d1e_e7eb1ea7600a.slice/crio-0af2b6abc6ed57a754be68eaaa83de04d14de13bd36b44b30c4ee443102f1772 WatchSource:0}: Error finding container 0af2b6abc6ed57a754be68eaaa83de04d14de13bd36b44b30c4ee443102f1772: Status 404 returned error can't find the container with id 0af2b6abc6ed57a754be68eaaa83de04d14de13bd36b44b30c4ee443102f1772
Oct 30 13:18:13.936704 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:13.936659    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:18:13.936802 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:13.936755    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:18:13.936843 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:13.936803    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:18:14.690567 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:14.690530    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls" event={"ID":"c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a","Type":"ContainerStarted","Data":"70fd6e41d90aac4da67c626477432abffab291b4d1c96fa15e5a27cec60480d5"}
Oct 30 13:18:14.693006 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:14.690588    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls" event={"ID":"c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a","Type":"ContainerStarted","Data":"0af2b6abc6ed57a754be68eaaa83de04d14de13bd36b44b30c4ee443102f1772"}
Oct 30 13:18:15.609654 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:15.609609    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls"
Oct 30 13:18:15.611374 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:15.611344    5093 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 13:18:15.611513 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:15.611398    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls" podUID="c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 13:18:16.609493 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:16.609453    5093 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 13:18:16.609905 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:16.609513    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls" podUID="c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 13:18:17.622155 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:17.618888    5093 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 13:18:17.622155 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:17.618975    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls" podUID="c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 13:18:18.620157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:18.619753    5093 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 13:18:18.620157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:18.619841    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls" podUID="c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 13:18:19.032184 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:19.021475    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt" podUID="4ae52c69-0c30-438f-9ac0-c87b6de16061" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 30 13:18:19.620952 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:19.620906    5093 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 13:18:19.621288 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:19.621248    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls" podUID="c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 13:18:19.677283 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:19.677240    5093 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:18:19.677579 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:19.677545    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn" podUID="de03af69-2ff9-4add-a940-740935198076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:19.724157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:19.723183    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv" podUID="d6a44981-104f-4d3e-9937-d184a6ddc482" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 13:18:19.797222 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:19.797083    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw" podUID="ad37954e-1350-4f75-a63f-a0c0f29ae416" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:19.802335 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:19.802280    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx" podUID="ba156eeb-9955-4f9a-857b-2c4d502a5fcc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:20.039181 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:20.037498    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt" podUID="4ae52c69-0c30-438f-9ac0-c87b6de16061" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:20.039181 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:20.037969    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt"
Oct 30 13:18:20.165080 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:20.165006    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls" podStartSLOduration=7.164987432 podStartE2EDuration="7.164987432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:14.766093008 +0000 UTC m=+913.913360251" watchObservedRunningTime="2026-10-30 13:18:20.164987432 +0000 UTC m=+919.312254674"
Oct 30 13:18:20.317688 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:20.292447    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25" podUID="7805f89f-89d4-4754-b606-c3070f4df215" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 13:18:20.620225 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:20.606386    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb" podUID="c4b4e00e-f58d-4a4e-b7db-19f724360db1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:20.620574 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:20.620520    5093 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25" podUID="7805f89f-89d4-4754-b606-c3070f4df215" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:20.635837 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:20.635787    5093 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 13:18:20.636208 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:20.636166    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls" podUID="c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 13:18:20.763478 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:20.734981    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv" podUID="d6a44981-104f-4d3e-9937-d184a6ddc482" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:20.766376 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:20.766343    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv"
Oct 30 13:18:20.772198 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:20.743590    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg" podUID="ea079ac1-f502-4b5f-b16a-c91e2de98de8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Oct 30 13:18:20.813580 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:20.813523    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw" podUID="ad37954e-1350-4f75-a63f-a0c0f29ae416" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:20.838574 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:20.821303    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx" podUID="ba156eeb-9955-4f9a-857b-2c4d502a5fcc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 30 13:18:20.892324 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:20.875323    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw"
Oct 30 13:18:20.892324 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:20.875405    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx"
Oct 30 13:18:21.099725 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:21.048530    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt" podUID="4ae52c69-0c30-438f-9ac0-c87b6de16061" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 30 13:18:21.315602 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:21.313633    5093 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt" podUID="4ae52c69-0c30-438f-9ac0-c87b6de16061" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:21.492730 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:21.492673    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25" podUID="7805f89f-89d4-4754-b606-c3070f4df215" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:21.521617 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:21.521588    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25"
Oct 30 13:18:21.640608 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:21.640505    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb" podUID="c4b4e00e-f58d-4a4e-b7db-19f724360db1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Oct 30 13:18:21.642377 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:21.642351    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb"
Oct 30 13:18:21.683204 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:21.645001    5093 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 13:18:21.683640 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:21.683601    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls" podUID="c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 13:18:21.803199 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:21.779495    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv" podUID="d6a44981-104f-4d3e-9937-d184a6ddc482" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 13:18:21.808201 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:21.806937    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg" podUID="ea079ac1-f502-4b5f-b16a-c91e2de98de8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:21.822188 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:21.808476    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg"
Oct 30 13:18:21.859156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:21.855296    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx" podUID="ba156eeb-9955-4f9a-857b-2c4d502a5fcc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:21.859156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:21.858668    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw" podUID="ad37954e-1350-4f75-a63f-a0c0f29ae416" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 13:18:22.073284 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:22.073233    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt" podUID="4ae52c69-0c30-438f-9ac0-c87b6de16061" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:22.366454 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:18:22.366357    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:18:22.476257 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:22.452080    5093 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw" podUID="ad37954e-1350-4f75-a63f-a0c0f29ae416" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": context deadline exceeded"
Oct 30 13:18:22.479920 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:22.478577    5093 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:18:22.481012 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:22.480966    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms" podUID="616525f4-fcee-4d8e-977e-f2104a46208f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:22.502750 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:22.502698    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25" podUID="7805f89f-89d4-4754-b606-c3070f4df215" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 13:18:22.611493 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:22.611455    5093 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 13:18:22.611780 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:22.611749    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls" podUID="c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 13:18:22.653484 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:22.653374    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb" podUID="c4b4e00e-f58d-4a4e-b7db-19f724360db1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Oct 30 13:18:22.783991 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:22.783941    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv" podUID="d6a44981-104f-4d3e-9937-d184a6ddc482" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 13:18:22.809039 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:22.808268    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg" podUID="ea079ac1-f502-4b5f-b16a-c91e2de98de8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:22.862625 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:22.861519    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx" podUID="ba156eeb-9955-4f9a-857b-2c4d502a5fcc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 30 13:18:22.912965 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:22.904261    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw" podUID="ad37954e-1350-4f75-a63f-a0c0f29ae416" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:22.955458 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:22.955421    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg"
Oct 30 13:18:22.960751 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:22.960720    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv"
Oct 30 13:18:22.974417 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:22.974387    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx"
Oct 30 13:18:22.984306 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:22.984275    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25"
Oct 30 13:18:23.026449 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:23.026416    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb"
Oct 30 13:18:23.026731 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:23.026716    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt"
Oct 30 13:18:23.088558 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:23.088511    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw"
Oct 30 13:18:23.377631 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:23.377594    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj"]
Oct 30 13:18:23.383550 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:23.383522    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj"
Oct 30 13:18:23.388519 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:23.388493    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj"]
Oct 30 13:18:23.414297 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:23.414263    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-credential-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj\" (UID: \"f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj"
Oct 30 13:18:23.414486 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:23.414326    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbmdq\" (UniqueName: \"kubernetes.io/projected/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-kube-api-access-tbmdq\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj\" (UID: \"f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj"
Oct 30 13:18:23.414486 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:23.414369    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-workload-certs\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj\" (UID: \"f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj"
Oct 30 13:18:23.414486 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:23.414455    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-istiod-ca-cert\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj\" (UID: \"f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj"
Oct 30 13:18:23.414654 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:23.414534    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-workload-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj\" (UID: \"f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj"
Oct 30 13:18:23.414654 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:23.414589    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-istio-envoy\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj\" (UID: \"f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj"
Oct 30 13:18:23.414654 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:23.414613    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-istio-data\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj\" (UID: \"f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj"
Oct 30 13:18:23.414816 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:23.414653    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-istio-token\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj\" (UID: \"f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj"
Oct 30 13:18:23.414816 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:23.414697    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-istio-podinfo\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj\" (UID: \"f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj"
Oct 30 13:18:23.516050 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:23.516006    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-credential-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj\" (UID: \"f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj"
Oct 30 13:18:23.516273 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:23.516064    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbmdq\" (UniqueName: \"kubernetes.io/projected/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-kube-api-access-tbmdq\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj\" (UID: \"f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj"
Oct 30 13:18:23.516273 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:23.516101    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-workload-certs\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj\" (UID: \"f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj"
Oct 30 13:18:23.516273 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:23.516188    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-istiod-ca-cert\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj\" (UID: \"f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj"
Oct 30 13:18:23.516273 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:23.516262    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-workload-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj\" (UID: \"f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj"
Oct 30 13:18:23.516596 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:23.516317    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-istio-envoy\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj\" (UID: \"f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj"
Oct 30 13:18:23.516596 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:23.516340    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-istio-data\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj\" (UID: \"f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj"
Oct 30 13:18:23.516596 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:23.516380    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-istio-token\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj\" (UID: \"f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj"
Oct 30 13:18:23.516596 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:23.516405    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-credential-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj\" (UID: \"f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj"
Oct 30 13:18:23.516596 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:23.516424    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-istio-podinfo\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj\" (UID: \"f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj"
Oct 30 13:18:23.516596 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:23.516584    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-workload-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj\" (UID: \"f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj"
Oct 30 13:18:23.516920 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:23.516886    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-istio-data\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj\" (UID: \"f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj"
Oct 30 13:18:23.517247 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:23.517226    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-workload-certs\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj\" (UID: \"f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj"
Oct 30 13:18:23.517344 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:23.517263    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-istiod-ca-cert\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj\" (UID: \"f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj"
Oct 30 13:18:23.519099 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:23.519079    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-istio-envoy\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj\" (UID: \"f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj"
Oct 30 13:18:23.519309 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:23.519289    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-istio-podinfo\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj\" (UID: \"f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj"
Oct 30 13:18:23.541358 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:23.541321    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-istio-token\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj\" (UID: \"f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj"
Oct 30 13:18:23.542750 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:23.542717    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbmdq\" (UniqueName: \"kubernetes.io/projected/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-kube-api-access-tbmdq\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj\" (UID: \"f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj"
Oct 30 13:18:23.609634 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:23.609588    5093 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 13:18:23.609858 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:23.609649    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls" podUID="c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 13:18:23.708379 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:23.708330    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj"
Oct 30 13:18:24.040632 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:24.040601    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj"]
Oct 30 13:18:24.042336 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:18:24.042301    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3c6ad8f_cf78_4dfb_8a99_a9947fe676f4.slice/crio-002a7dd3a4c32ac93e4eeed42007ae5f26d53ee4f9ac82662b5a4a08e6b2c1bc WatchSource:0}: Error finding container 002a7dd3a4c32ac93e4eeed42007ae5f26d53ee4f9ac82662b5a4a08e6b2c1bc: Status 404 returned error can't find the container with id 002a7dd3a4c32ac93e4eeed42007ae5f26d53ee4f9ac82662b5a4a08e6b2c1bc
Oct 30 13:18:24.045775 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:24.045735    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:18:24.045902 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:24.045826    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:18:24.045902 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:24.045878    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:18:24.504222 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:24.504172    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj" event={"ID":"f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4","Type":"ContainerStarted","Data":"95ed86f855f248c3cd33894964957b249b06254dd5892c320941bc46a8c1a29e"}
Oct 30 13:18:24.504663 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:24.504249    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj" event={"ID":"f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4","Type":"ContainerStarted","Data":"002a7dd3a4c32ac93e4eeed42007ae5f26d53ee4f9ac82662b5a4a08e6b2c1bc"}
Oct 30 13:18:24.550345 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:24.550281    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj" podStartSLOduration=1.550265684 podStartE2EDuration="1.550265684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:24.548590288 +0000 UTC m=+923.695857530" watchObservedRunningTime="2026-10-30 13:18:24.550265684 +0000 UTC m=+923.697532925"
Oct 30 13:18:24.709369 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:24.709330    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj"
Oct 30 13:18:25.092863 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:25.092763    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls"
Oct 30 13:18:25.093185 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:25.093158    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls"
Oct 30 13:18:25.094099 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:25.094080    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls"
Oct 30 13:18:25.710378 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:25.710341    5093 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 13:18:25.710828 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:25.710396    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj" podUID="f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 13:18:26.716599 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:26.709124    5093 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 13:18:26.716599 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:26.709215    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj" podUID="f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 13:18:27.708990 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:27.708949    5093 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 13:18:27.709202 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:27.709011    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj" podUID="f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 13:18:28.322359 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:28.322324    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9"]
Oct 30 13:18:28.342798 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:28.342764    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9"
Oct 30 13:18:28.363205 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:28.363167    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9"]
Oct 30 13:18:28.484759 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:28.484723    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-workload-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9\" (UID: \"1b9ce40f-35d8-4041-a510-37f2cd68d2b3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9"
Oct 30 13:18:28.484759 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:28.484761    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-istio-podinfo\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9\" (UID: \"1b9ce40f-35d8-4041-a510-37f2cd68d2b3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9"
Oct 30 13:18:28.485001 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:28.484789    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-istio-data\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9\" (UID: \"1b9ce40f-35d8-4041-a510-37f2cd68d2b3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9"
Oct 30 13:18:28.485001 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:28.484862    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-istio-token\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9\" (UID: \"1b9ce40f-35d8-4041-a510-37f2cd68d2b3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9"
Oct 30 13:18:28.485082 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:28.485058    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-istio-envoy\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9\" (UID: \"1b9ce40f-35d8-4041-a510-37f2cd68d2b3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9"
Oct 30 13:18:28.485167 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:28.485091    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6k2c\" (UniqueName: \"kubernetes.io/projected/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-kube-api-access-q6k2c\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9\" (UID: \"1b9ce40f-35d8-4041-a510-37f2cd68d2b3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9"
Oct 30 13:18:28.485391 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:28.485367    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-credential-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9\" (UID: \"1b9ce40f-35d8-4041-a510-37f2cd68d2b3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9"
Oct 30 13:18:28.485431 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:28.485410    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-workload-certs\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9\" (UID: \"1b9ce40f-35d8-4041-a510-37f2cd68d2b3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9"
Oct 30 13:18:28.485517 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:28.485497    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-istiod-ca-cert\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9\" (UID: \"1b9ce40f-35d8-4041-a510-37f2cd68d2b3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9"
Oct 30 13:18:28.586477 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:28.586386    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-credential-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9\" (UID: \"1b9ce40f-35d8-4041-a510-37f2cd68d2b3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9"
Oct 30 13:18:28.586477 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:28.586428    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-workload-certs\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9\" (UID: \"1b9ce40f-35d8-4041-a510-37f2cd68d2b3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9"
Oct 30 13:18:28.586477 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:28.586472    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-istiod-ca-cert\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9\" (UID: \"1b9ce40f-35d8-4041-a510-37f2cd68d2b3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9"
Oct 30 13:18:28.586761 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:28.586517    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-workload-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9\" (UID: \"1b9ce40f-35d8-4041-a510-37f2cd68d2b3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9"
Oct 30 13:18:28.586761 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:28.586545    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-istio-podinfo\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9\" (UID: \"1b9ce40f-35d8-4041-a510-37f2cd68d2b3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9"
Oct 30 13:18:28.586761 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:28.586581    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-istio-data\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9\" (UID: \"1b9ce40f-35d8-4041-a510-37f2cd68d2b3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9"
Oct 30 13:18:28.586761 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:28.586602    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-istio-token\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9\" (UID: \"1b9ce40f-35d8-4041-a510-37f2cd68d2b3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9"
Oct 30 13:18:28.586761 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:28.586691    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-istio-envoy\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9\" (UID: \"1b9ce40f-35d8-4041-a510-37f2cd68d2b3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9"
Oct 30 13:18:28.586761 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:28.586717    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6k2c\" (UniqueName: \"kubernetes.io/projected/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-kube-api-access-q6k2c\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9\" (UID: \"1b9ce40f-35d8-4041-a510-37f2cd68d2b3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9"
Oct 30 13:18:28.587293 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:28.587268    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-credential-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9\" (UID: \"1b9ce40f-35d8-4041-a510-37f2cd68d2b3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9"
Oct 30 13:18:28.587395 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:28.587335    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-istiod-ca-cert\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9\" (UID: \"1b9ce40f-35d8-4041-a510-37f2cd68d2b3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9"
Oct 30 13:18:28.587533 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:28.587509    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-workload-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9\" (UID: \"1b9ce40f-35d8-4041-a510-37f2cd68d2b3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9"
Oct 30 13:18:28.587649 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:28.587631    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-istio-data\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9\" (UID: \"1b9ce40f-35d8-4041-a510-37f2cd68d2b3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9"
Oct 30 13:18:28.587872 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:28.587854    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-workload-certs\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9\" (UID: \"1b9ce40f-35d8-4041-a510-37f2cd68d2b3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9"
Oct 30 13:18:28.589303 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:28.589263    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-istio-podinfo\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9\" (UID: \"1b9ce40f-35d8-4041-a510-37f2cd68d2b3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9"
Oct 30 13:18:28.589440 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:28.589420    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-istio-envoy\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9\" (UID: \"1b9ce40f-35d8-4041-a510-37f2cd68d2b3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9"
Oct 30 13:18:28.607054 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:28.607023    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-istio-token\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9\" (UID: \"1b9ce40f-35d8-4041-a510-37f2cd68d2b3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9"
Oct 30 13:18:28.607222 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:28.607207    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6k2c\" (UniqueName: \"kubernetes.io/projected/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-kube-api-access-q6k2c\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9\" (UID: \"1b9ce40f-35d8-4041-a510-37f2cd68d2b3\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9"
Oct 30 13:18:28.673522 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:28.673482    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9"
Oct 30 13:18:28.712158 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:28.708787    5093 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 13:18:28.712158 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:28.708851    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj" podUID="f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 13:18:29.063508 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:29.060896    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9"]
Oct 30 13:18:29.069594 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:29.069549    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:18:29.069728 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:29.069660    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:18:29.069791 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:29.069740    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:18:29.637156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:29.630089    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9" event={"ID":"1b9ce40f-35d8-4041-a510-37f2cd68d2b3","Type":"ContainerStarted","Data":"e149248c2ec9bee6cc4de580d5fada481c55f798eca11f4b6e88948a8955c455"}
Oct 30 13:18:29.637156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:29.630185    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9" event={"ID":"1b9ce40f-35d8-4041-a510-37f2cd68d2b3","Type":"ContainerStarted","Data":"e0b879742234f48543163c3aed7b59b91ab64e9e12390a48b22271d5caeb7ffb"}
Oct 30 13:18:29.677854 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:29.672625    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf"]
Oct 30 13:18:29.709688 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:29.709657    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9"
Oct 30 13:18:29.709905 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:29.709890    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf"]
Oct 30 13:18:29.711413 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:29.711391    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf"
Oct 30 13:18:29.714360 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:29.714165    5093 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 13:18:29.714360 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:29.714218    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj" podUID="f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 13:18:29.717535 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:29.717472    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9" podStartSLOduration=1.717451856 podStartE2EDuration="1.717451856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:29.697963541 +0000 UTC m=+928.845230783" watchObservedRunningTime="2026-10-30 13:18:29.717451856 +0000 UTC m=+928.864719101"
Oct 30 13:18:29.739570 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:29.739538    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-6b7b93a3-openshift-default-dockercfg-42stm\""
Oct 30 13:18:29.844194 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:29.817356    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/528a3a61-adc9-45fd-aefd-1cc67d8cf716-istio-podinfo\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf\" (UID: \"528a3a61-adc9-45fd-aefd-1cc67d8cf716\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf"
Oct 30 13:18:29.853301 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:29.853259    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/528a3a61-adc9-45fd-aefd-1cc67d8cf716-workload-socket\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf\" (UID: \"528a3a61-adc9-45fd-aefd-1cc67d8cf716\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf"
Oct 30 13:18:29.861351 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:29.861303    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/528a3a61-adc9-45fd-aefd-1cc67d8cf716-istio-token\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf\" (UID: \"528a3a61-adc9-45fd-aefd-1cc67d8cf716\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf"
Oct 30 13:18:29.861640 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:29.861618    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7kv9\" (UniqueName: \"kubernetes.io/projected/528a3a61-adc9-45fd-aefd-1cc67d8cf716-kube-api-access-p7kv9\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf\" (UID: \"528a3a61-adc9-45fd-aefd-1cc67d8cf716\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf"
Oct 30 13:18:29.874344 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:29.874297    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/528a3a61-adc9-45fd-aefd-1cc67d8cf716-istio-envoy\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf\" (UID: \"528a3a61-adc9-45fd-aefd-1cc67d8cf716\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf"
Oct 30 13:18:29.874640 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:29.874621    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/528a3a61-adc9-45fd-aefd-1cc67d8cf716-credential-socket\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf\" (UID: \"528a3a61-adc9-45fd-aefd-1cc67d8cf716\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf"
Oct 30 13:18:29.874855 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:29.874838    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/528a3a61-adc9-45fd-aefd-1cc67d8cf716-istiod-ca-cert\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf\" (UID: \"528a3a61-adc9-45fd-aefd-1cc67d8cf716\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf"
Oct 30 13:18:29.874999 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:29.874984    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/528a3a61-adc9-45fd-aefd-1cc67d8cf716-workload-certs\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf\" (UID: \"528a3a61-adc9-45fd-aefd-1cc67d8cf716\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf"
Oct 30 13:18:29.889461 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:29.875160    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/528a3a61-adc9-45fd-aefd-1cc67d8cf716-istio-data\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf\" (UID: \"528a3a61-adc9-45fd-aefd-1cc67d8cf716\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf"
Oct 30 13:18:30.012365 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:30.002626    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/528a3a61-adc9-45fd-aefd-1cc67d8cf716-istio-token\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf\" (UID: \"528a3a61-adc9-45fd-aefd-1cc67d8cf716\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf"
Oct 30 13:18:30.012710 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:30.012680    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7kv9\" (UniqueName: \"kubernetes.io/projected/528a3a61-adc9-45fd-aefd-1cc67d8cf716-kube-api-access-p7kv9\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf\" (UID: \"528a3a61-adc9-45fd-aefd-1cc67d8cf716\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf"
Oct 30 13:18:30.012921 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:30.012906    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/528a3a61-adc9-45fd-aefd-1cc67d8cf716-istio-envoy\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf\" (UID: \"528a3a61-adc9-45fd-aefd-1cc67d8cf716\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf"
Oct 30 13:18:30.013061 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:30.013047    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/528a3a61-adc9-45fd-aefd-1cc67d8cf716-credential-socket\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf\" (UID: \"528a3a61-adc9-45fd-aefd-1cc67d8cf716\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf"
Oct 30 13:18:30.013234 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:30.013218    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/528a3a61-adc9-45fd-aefd-1cc67d8cf716-istiod-ca-cert\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf\" (UID: \"528a3a61-adc9-45fd-aefd-1cc67d8cf716\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf"
Oct 30 13:18:30.013414 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:30.013396    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/528a3a61-adc9-45fd-aefd-1cc67d8cf716-workload-certs\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf\" (UID: \"528a3a61-adc9-45fd-aefd-1cc67d8cf716\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf"
Oct 30 13:18:30.013587 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:30.013571    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/528a3a61-adc9-45fd-aefd-1cc67d8cf716-istio-data\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf\" (UID: \"528a3a61-adc9-45fd-aefd-1cc67d8cf716\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf"
Oct 30 13:18:30.013739 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:30.013726    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/528a3a61-adc9-45fd-aefd-1cc67d8cf716-istio-podinfo\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf\" (UID: \"528a3a61-adc9-45fd-aefd-1cc67d8cf716\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf"
Oct 30 13:18:30.013849 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:30.013833    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/528a3a61-adc9-45fd-aefd-1cc67d8cf716-workload-socket\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf\" (UID: \"528a3a61-adc9-45fd-aefd-1cc67d8cf716\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf"
Oct 30 13:18:30.014488 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:30.014463    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/528a3a61-adc9-45fd-aefd-1cc67d8cf716-workload-socket\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf\" (UID: \"528a3a61-adc9-45fd-aefd-1cc67d8cf716\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf"
Oct 30 13:18:30.017511 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:30.017487    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/528a3a61-adc9-45fd-aefd-1cc67d8cf716-workload-certs\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf\" (UID: \"528a3a61-adc9-45fd-aefd-1cc67d8cf716\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf"
Oct 30 13:18:30.017923 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:30.017900    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/528a3a61-adc9-45fd-aefd-1cc67d8cf716-credential-socket\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf\" (UID: \"528a3a61-adc9-45fd-aefd-1cc67d8cf716\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf"
Oct 30 13:18:30.018710 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:30.018686    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/528a3a61-adc9-45fd-aefd-1cc67d8cf716-istiod-ca-cert\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf\" (UID: \"528a3a61-adc9-45fd-aefd-1cc67d8cf716\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf"
Oct 30 13:18:30.019086 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:30.019068    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/528a3a61-adc9-45fd-aefd-1cc67d8cf716-istio-data\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf\" (UID: \"528a3a61-adc9-45fd-aefd-1cc67d8cf716\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf"
Oct 30 13:18:30.062782 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:30.062740    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/528a3a61-adc9-45fd-aefd-1cc67d8cf716-istio-token\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf\" (UID: \"528a3a61-adc9-45fd-aefd-1cc67d8cf716\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf"
Oct 30 13:18:30.079531 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:30.076082    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7kv9\" (UniqueName: \"kubernetes.io/projected/528a3a61-adc9-45fd-aefd-1cc67d8cf716-kube-api-access-p7kv9\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf\" (UID: \"528a3a61-adc9-45fd-aefd-1cc67d8cf716\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf"
Oct 30 13:18:30.086150 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:30.080920    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/528a3a61-adc9-45fd-aefd-1cc67d8cf716-istio-envoy\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf\" (UID: \"528a3a61-adc9-45fd-aefd-1cc67d8cf716\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf"
Oct 30 13:18:30.086150 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:30.083379    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/528a3a61-adc9-45fd-aefd-1cc67d8cf716-istio-podinfo\") pod \"e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf\" (UID: \"528a3a61-adc9-45fd-aefd-1cc67d8cf716\") " pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf"
Oct 30 13:18:30.214421 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:30.214149    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf"
Oct 30 13:18:30.675623 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:30.675588    5093 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 13:18:30.676255 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:30.675648    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9" podUID="1b9ce40f-35d8-4041-a510-37f2cd68d2b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 13:18:30.685871 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:30.685837    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf"]
Oct 30 13:18:30.692111 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:30.692075    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:18:30.692231 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:30.692212    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:18:30.692296 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:30.692282    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:18:30.709227 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:30.709193    5093 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 13:18:30.709444 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:30.709416    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj" podUID="f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 13:18:31.675194 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:31.674260    5093 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 13:18:31.675194 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:31.674340    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9" podUID="1b9ce40f-35d8-4041-a510-37f2cd68d2b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 13:18:31.681930 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:31.681882    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf" event={"ID":"528a3a61-adc9-45fd-aefd-1cc67d8cf716","Type":"ContainerStarted","Data":"c7944a7e350c9ad5a27bc5f7a09cabd57d9ad46fdb7ec8104d2a6c7186d462d4"}
Oct 30 13:18:31.682470 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:31.681961    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf" event={"ID":"528a3a61-adc9-45fd-aefd-1cc67d8cf716","Type":"ContainerStarted","Data":"32b7fc50db424eba404bb56363cc1f0b5cfd4bb1edc5f00d40aac94e4fd8b3a1"}
Oct 30 13:18:32.215291 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:32.215123    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf"
Oct 30 13:18:32.220249 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:32.220215    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf"
Oct 30 13:18:32.276180 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:32.274457    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf" podStartSLOduration=3.27444055 podStartE2EDuration="3.27444055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:31.747506175 +0000 UTC m=+930.894773441" watchObservedRunningTime="2026-10-30 13:18:32.27444055 +0000 UTC m=+931.421707790"
Oct 30 13:18:32.311983 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:32.311901    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj"
Oct 30 13:18:32.312994 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:32.312967    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj"
Oct 30 13:18:32.314051 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:32.314028    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj"
Oct 30 13:18:32.687156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:32.674686    5093 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 13:18:32.687156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:32.674767    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9" podUID="1b9ce40f-35d8-4041-a510-37f2cd68d2b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 13:18:32.709583 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:32.709522    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf"
Oct 30 13:18:32.711741 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:32.711483    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf"
Oct 30 13:18:33.674092 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:33.674049    5093 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 13:18:33.674292 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:33.674122    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9" podUID="1b9ce40f-35d8-4041-a510-37f2cd68d2b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 13:18:34.674491 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:34.674447    5093 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 13:18:34.674905 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:34.674519    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9" podUID="1b9ce40f-35d8-4041-a510-37f2cd68d2b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 13:18:35.674998 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:35.674953    5093 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 13:18:35.675611 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:35.675020    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9" podUID="1b9ce40f-35d8-4041-a510-37f2cd68d2b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 13:18:35.783946 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:35.783907    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx"]
Oct 30 13:18:35.788849 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:35.788685    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx"
Oct 30 13:18:35.817890 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:35.817858    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-istio-envoy\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx\" (UID: \"67c19bdd-7c73-46e7-8e2b-58eef9290ce0\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx"
Oct 30 13:18:35.818146 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:35.817904    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdcbl\" (UniqueName: \"kubernetes.io/projected/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-kube-api-access-sdcbl\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx\" (UID: \"67c19bdd-7c73-46e7-8e2b-58eef9290ce0\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx"
Oct 30 13:18:35.818146 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:35.817925    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-istiod-ca-cert\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx\" (UID: \"67c19bdd-7c73-46e7-8e2b-58eef9290ce0\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx"
Oct 30 13:18:35.818146 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:35.817965    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-istio-podinfo\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx\" (UID: \"67c19bdd-7c73-46e7-8e2b-58eef9290ce0\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx"
Oct 30 13:18:35.818146 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:35.817984    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-credential-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx\" (UID: \"67c19bdd-7c73-46e7-8e2b-58eef9290ce0\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx"
Oct 30 13:18:35.818146 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:35.818051    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-workload-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx\" (UID: \"67c19bdd-7c73-46e7-8e2b-58eef9290ce0\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx"
Oct 30 13:18:35.818146 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:35.818093    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-istio-data\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx\" (UID: \"67c19bdd-7c73-46e7-8e2b-58eef9290ce0\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx"
Oct 30 13:18:35.818577 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:35.818173    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-workload-certs\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx\" (UID: \"67c19bdd-7c73-46e7-8e2b-58eef9290ce0\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx"
Oct 30 13:18:35.818577 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:35.818206    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-istio-token\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx\" (UID: \"67c19bdd-7c73-46e7-8e2b-58eef9290ce0\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx"
Oct 30 13:18:35.821485 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:35.821435    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx"]
Oct 30 13:18:35.919883 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:35.919837    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-istio-token\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx\" (UID: \"67c19bdd-7c73-46e7-8e2b-58eef9290ce0\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx"
Oct 30 13:18:35.920063 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:35.920015    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-istio-envoy\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx\" (UID: \"67c19bdd-7c73-46e7-8e2b-58eef9290ce0\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx"
Oct 30 13:18:35.920177 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:35.920064    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdcbl\" (UniqueName: \"kubernetes.io/projected/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-kube-api-access-sdcbl\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx\" (UID: \"67c19bdd-7c73-46e7-8e2b-58eef9290ce0\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx"
Oct 30 13:18:35.920177 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:35.920094    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-istiod-ca-cert\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx\" (UID: \"67c19bdd-7c73-46e7-8e2b-58eef9290ce0\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx"
Oct 30 13:18:35.920408 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:35.920172    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-istio-podinfo\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx\" (UID: \"67c19bdd-7c73-46e7-8e2b-58eef9290ce0\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx"
Oct 30 13:18:35.920408 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:35.920206    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-credential-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx\" (UID: \"67c19bdd-7c73-46e7-8e2b-58eef9290ce0\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx"
Oct 30 13:18:35.920702 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:35.920673    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-credential-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx\" (UID: \"67c19bdd-7c73-46e7-8e2b-58eef9290ce0\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx"
Oct 30 13:18:35.920830 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:35.920803    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-workload-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx\" (UID: \"67c19bdd-7c73-46e7-8e2b-58eef9290ce0\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx"
Oct 30 13:18:35.920896 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:35.920877    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-istio-data\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx\" (UID: \"67c19bdd-7c73-46e7-8e2b-58eef9290ce0\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx"
Oct 30 13:18:35.920954 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:35.920888    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-istiod-ca-cert\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx\" (UID: \"67c19bdd-7c73-46e7-8e2b-58eef9290ce0\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx"
Oct 30 13:18:35.921015 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:35.920990    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-workload-certs\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx\" (UID: \"67c19bdd-7c73-46e7-8e2b-58eef9290ce0\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx"
Oct 30 13:18:35.921317 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:35.921286    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-workload-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx\" (UID: \"67c19bdd-7c73-46e7-8e2b-58eef9290ce0\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx"
Oct 30 13:18:35.921415 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:35.921344    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-istio-data\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx\" (UID: \"67c19bdd-7c73-46e7-8e2b-58eef9290ce0\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx"
Oct 30 13:18:35.921491 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:35.921407    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-workload-certs\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx\" (UID: \"67c19bdd-7c73-46e7-8e2b-58eef9290ce0\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx"
Oct 30 13:18:35.922703 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:35.922683    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-istio-envoy\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx\" (UID: \"67c19bdd-7c73-46e7-8e2b-58eef9290ce0\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx"
Oct 30 13:18:35.922956 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:35.922934    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-istio-podinfo\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx\" (UID: \"67c19bdd-7c73-46e7-8e2b-58eef9290ce0\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx"
Oct 30 13:18:35.941603 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:35.941514    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-istio-token\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx\" (UID: \"67c19bdd-7c73-46e7-8e2b-58eef9290ce0\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx"
Oct 30 13:18:35.941851 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:35.941820    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdcbl\" (UniqueName: \"kubernetes.io/projected/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-kube-api-access-sdcbl\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx\" (UID: \"67c19bdd-7c73-46e7-8e2b-58eef9290ce0\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx"
Oct 30 13:18:36.132192 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:36.132157    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx"
Oct 30 13:18:36.563864 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:36.563828    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx"]
Oct 30 13:18:36.565669 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:18:36.565635    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67c19bdd_7c73_46e7_8e2b_58eef9290ce0.slice/crio-8015f3c8762822c688c963ba227d9c58b9314af0c08527d78c5b00f930a64536 WatchSource:0}: Error finding container 8015f3c8762822c688c963ba227d9c58b9314af0c08527d78c5b00f930a64536: Status 404 returned error can't find the container with id 8015f3c8762822c688c963ba227d9c58b9314af0c08527d78c5b00f930a64536
Oct 30 13:18:36.568702 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:36.568666    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:18:36.568813 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:36.568737    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:18:36.568813 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:36.568765    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:18:36.765463 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:36.765430    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx" event={"ID":"67c19bdd-7c73-46e7-8e2b-58eef9290ce0","Type":"ContainerStarted","Data":"2dd023c3100090caf4c69c1ddb7616bfd617ff2af54ed1ee8f05d7c0224f088c"}
Oct 30 13:18:36.765941 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:36.765919    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx" event={"ID":"67c19bdd-7c73-46e7-8e2b-58eef9290ce0","Type":"ContainerStarted","Data":"8015f3c8762822c688c963ba227d9c58b9314af0c08527d78c5b00f930a64536"}
Oct 30 13:18:36.820349 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:36.820194    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx" podStartSLOduration=1.820170772 podStartE2EDuration="1.820170772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:36.812699449 +0000 UTC m=+935.959966690" watchObservedRunningTime="2026-10-30 13:18:36.820170772 +0000 UTC m=+935.967438014"
Oct 30 13:18:37.143990 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:37.141289    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx"
Oct 30 13:18:37.146260 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:37.146223    5093 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 13:18:37.146515 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:37.146484    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx" podUID="67c19bdd-7c73-46e7-8e2b-58eef9290ce0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 13:18:37.288081 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:37.288051    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9"
Oct 30 13:18:37.288605 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:37.288577    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9"
Oct 30 13:18:37.289816 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:37.289785    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9"
Oct 30 13:18:38.150154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:38.149598    5093 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 13:18:38.150154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:38.149666    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx" podUID="67c19bdd-7c73-46e7-8e2b-58eef9290ce0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 13:18:38.760158 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:38.756598    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn"]
Oct 30 13:18:39.085944 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.085802    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn"
Oct 30 13:18:39.154519 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.143619    5093 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 13:18:39.154519 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.143675    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx" podUID="67c19bdd-7c73-46e7-8e2b-58eef9290ce0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 13:18:39.265156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.257690    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/874dabda-7e3e-44eb-9768-7ae00d30d22e-istio-data\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn\" (UID: \"874dabda-7e3e-44eb-9768-7ae00d30d22e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn"
Oct 30 13:18:39.319969 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.319922    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/874dabda-7e3e-44eb-9768-7ae00d30d22e-workload-certs\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn\" (UID: \"874dabda-7e3e-44eb-9768-7ae00d30d22e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn"
Oct 30 13:18:39.371608 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.371497    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/874dabda-7e3e-44eb-9768-7ae00d30d22e-istio-podinfo\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn\" (UID: \"874dabda-7e3e-44eb-9768-7ae00d30d22e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn"
Oct 30 13:18:39.372860 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.372488    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94hq4\" (UniqueName: \"kubernetes.io/projected/874dabda-7e3e-44eb-9768-7ae00d30d22e-kube-api-access-94hq4\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn\" (UID: \"874dabda-7e3e-44eb-9768-7ae00d30d22e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn"
Oct 30 13:18:39.373496 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.373465    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/874dabda-7e3e-44eb-9768-7ae00d30d22e-istio-token\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn\" (UID: \"874dabda-7e3e-44eb-9768-7ae00d30d22e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn"
Oct 30 13:18:39.374259 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.374234    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/874dabda-7e3e-44eb-9768-7ae00d30d22e-istiod-ca-cert\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn\" (UID: \"874dabda-7e3e-44eb-9768-7ae00d30d22e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn"
Oct 30 13:18:39.374780 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.374753    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/874dabda-7e3e-44eb-9768-7ae00d30d22e-workload-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn\" (UID: \"874dabda-7e3e-44eb-9768-7ae00d30d22e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn"
Oct 30 13:18:39.389422 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.389385    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/874dabda-7e3e-44eb-9768-7ae00d30d22e-credential-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn\" (UID: \"874dabda-7e3e-44eb-9768-7ae00d30d22e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn"
Oct 30 13:18:39.390615 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.390588    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/874dabda-7e3e-44eb-9768-7ae00d30d22e-istio-envoy\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn\" (UID: \"874dabda-7e3e-44eb-9768-7ae00d30d22e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn"
Oct 30 13:18:39.438883 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.438847    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh"]
Oct 30 13:18:39.467436 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.467398    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn"]
Oct 30 13:18:39.467790 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.467769    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh"]
Oct 30 13:18:39.468990 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.468966    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh"
Oct 30 13:18:39.519284 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.519237    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/874dabda-7e3e-44eb-9768-7ae00d30d22e-istio-data\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn\" (UID: \"874dabda-7e3e-44eb-9768-7ae00d30d22e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn"
Oct 30 13:18:39.538262 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.519948    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/874dabda-7e3e-44eb-9768-7ae00d30d22e-istio-data\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn\" (UID: \"874dabda-7e3e-44eb-9768-7ae00d30d22e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn"
Oct 30 13:18:39.549583 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.548368    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/874dabda-7e3e-44eb-9768-7ae00d30d22e-workload-certs\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn\" (UID: \"874dabda-7e3e-44eb-9768-7ae00d30d22e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn"
Oct 30 13:18:39.549583 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.548445    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/874dabda-7e3e-44eb-9768-7ae00d30d22e-istio-podinfo\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn\" (UID: \"874dabda-7e3e-44eb-9768-7ae00d30d22e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn"
Oct 30 13:18:39.549583 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.548507    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94hq4\" (UniqueName: \"kubernetes.io/projected/874dabda-7e3e-44eb-9768-7ae00d30d22e-kube-api-access-94hq4\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn\" (UID: \"874dabda-7e3e-44eb-9768-7ae00d30d22e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn"
Oct 30 13:18:39.549583 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.548568    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/874dabda-7e3e-44eb-9768-7ae00d30d22e-istio-token\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn\" (UID: \"874dabda-7e3e-44eb-9768-7ae00d30d22e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn"
Oct 30 13:18:39.549583 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.548596    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/874dabda-7e3e-44eb-9768-7ae00d30d22e-istiod-ca-cert\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn\" (UID: \"874dabda-7e3e-44eb-9768-7ae00d30d22e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn"
Oct 30 13:18:39.549583 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.548625    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/874dabda-7e3e-44eb-9768-7ae00d30d22e-workload-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn\" (UID: \"874dabda-7e3e-44eb-9768-7ae00d30d22e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn"
Oct 30 13:18:39.549583 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.548663    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/874dabda-7e3e-44eb-9768-7ae00d30d22e-credential-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn\" (UID: \"874dabda-7e3e-44eb-9768-7ae00d30d22e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn"
Oct 30 13:18:39.549583 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.548744    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/874dabda-7e3e-44eb-9768-7ae00d30d22e-istio-envoy\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn\" (UID: \"874dabda-7e3e-44eb-9768-7ae00d30d22e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn"
Oct 30 13:18:39.550233 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.549830    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/874dabda-7e3e-44eb-9768-7ae00d30d22e-workload-certs\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn\" (UID: \"874dabda-7e3e-44eb-9768-7ae00d30d22e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn"
Oct 30 13:18:39.572799 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.572465    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/874dabda-7e3e-44eb-9768-7ae00d30d22e-workload-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn\" (UID: \"874dabda-7e3e-44eb-9768-7ae00d30d22e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn"
Oct 30 13:18:39.573314 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.573276    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/874dabda-7e3e-44eb-9768-7ae00d30d22e-credential-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn\" (UID: \"874dabda-7e3e-44eb-9768-7ae00d30d22e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn"
Oct 30 13:18:39.573969 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.573908    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/874dabda-7e3e-44eb-9768-7ae00d30d22e-istiod-ca-cert\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn\" (UID: \"874dabda-7e3e-44eb-9768-7ae00d30d22e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn"
Oct 30 13:18:39.630208 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.605035    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94hq4\" (UniqueName: \"kubernetes.io/projected/874dabda-7e3e-44eb-9768-7ae00d30d22e-kube-api-access-94hq4\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn\" (UID: \"874dabda-7e3e-44eb-9768-7ae00d30d22e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn"
Oct 30 13:18:39.656199 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.647859    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/874dabda-7e3e-44eb-9768-7ae00d30d22e-istio-podinfo\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn\" (UID: \"874dabda-7e3e-44eb-9768-7ae00d30d22e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn"
Oct 30 13:18:39.656482 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.652772    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/874dabda-7e3e-44eb-9768-7ae00d30d22e-istio-token\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn\" (UID: \"874dabda-7e3e-44eb-9768-7ae00d30d22e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn"
Oct 30 13:18:39.656613 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.653710    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt7sd\" (UniqueName: \"kubernetes.io/projected/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-kube-api-access-gt7sd\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh\" (UID: \"8bf396f4-4bbf-46dd-834e-2621e8c1f6cb\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh"
Oct 30 13:18:39.659293 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.659258    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-istio-envoy\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh\" (UID: \"8bf396f4-4bbf-46dd-834e-2621e8c1f6cb\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh"
Oct 30 13:18:39.659521 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.659501    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-istio-data\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh\" (UID: \"8bf396f4-4bbf-46dd-834e-2621e8c1f6cb\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh"
Oct 30 13:18:39.672875 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.672837    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-istiod-ca-cert\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh\" (UID: \"8bf396f4-4bbf-46dd-834e-2621e8c1f6cb\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh"
Oct 30 13:18:39.674085 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.674057    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-workload-certs\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh\" (UID: \"8bf396f4-4bbf-46dd-834e-2621e8c1f6cb\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh"
Oct 30 13:18:39.674406 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.674384    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-istio-podinfo\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh\" (UID: \"8bf396f4-4bbf-46dd-834e-2621e8c1f6cb\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh"
Oct 30 13:18:39.674571 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.674557    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-istio-token\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh\" (UID: \"8bf396f4-4bbf-46dd-834e-2621e8c1f6cb\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh"
Oct 30 13:18:39.674671 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.674658    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-workload-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh\" (UID: \"8bf396f4-4bbf-46dd-834e-2621e8c1f6cb\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh"
Oct 30 13:18:39.674818 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.674804    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-credential-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh\" (UID: \"8bf396f4-4bbf-46dd-834e-2621e8c1f6cb\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh"
Oct 30 13:18:39.767806 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.767733    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/874dabda-7e3e-44eb-9768-7ae00d30d22e-istio-envoy\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn\" (UID: \"874dabda-7e3e-44eb-9768-7ae00d30d22e\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn"
Oct 30 13:18:39.798398 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.778981    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-istio-token\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh\" (UID: \"8bf396f4-4bbf-46dd-834e-2621e8c1f6cb\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh"
Oct 30 13:18:39.798398 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.779029    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-workload-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh\" (UID: \"8bf396f4-4bbf-46dd-834e-2621e8c1f6cb\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh"
Oct 30 13:18:39.798398 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.779086    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-credential-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh\" (UID: \"8bf396f4-4bbf-46dd-834e-2621e8c1f6cb\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh"
Oct 30 13:18:39.798398 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.779198    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt7sd\" (UniqueName: \"kubernetes.io/projected/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-kube-api-access-gt7sd\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh\" (UID: \"8bf396f4-4bbf-46dd-834e-2621e8c1f6cb\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh"
Oct 30 13:18:39.798398 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.779250    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-istio-envoy\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh\" (UID: \"8bf396f4-4bbf-46dd-834e-2621e8c1f6cb\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh"
Oct 30 13:18:39.798398 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.779286    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-istio-data\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh\" (UID: \"8bf396f4-4bbf-46dd-834e-2621e8c1f6cb\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh"
Oct 30 13:18:39.798398 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.779344    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-istiod-ca-cert\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh\" (UID: \"8bf396f4-4bbf-46dd-834e-2621e8c1f6cb\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh"
Oct 30 13:18:39.798398 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.779457    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-workload-certs\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh\" (UID: \"8bf396f4-4bbf-46dd-834e-2621e8c1f6cb\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh"
Oct 30 13:18:39.798398 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.779523    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-istio-podinfo\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh\" (UID: \"8bf396f4-4bbf-46dd-834e-2621e8c1f6cb\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh"
Oct 30 13:18:39.819924 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.799829    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-istio-podinfo\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh\" (UID: \"8bf396f4-4bbf-46dd-834e-2621e8c1f6cb\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh"
Oct 30 13:18:39.820534 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.820499    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-istio-data\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh\" (UID: \"8bf396f4-4bbf-46dd-834e-2621e8c1f6cb\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh"
Oct 30 13:18:39.829082 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.829050    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-istiod-ca-cert\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh\" (UID: \"8bf396f4-4bbf-46dd-834e-2621e8c1f6cb\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh"
Oct 30 13:18:39.829514 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.829492    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-credential-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh\" (UID: \"8bf396f4-4bbf-46dd-834e-2621e8c1f6cb\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh"
Oct 30 13:18:39.833164 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.832464    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-workload-certs\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh\" (UID: \"8bf396f4-4bbf-46dd-834e-2621e8c1f6cb\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh"
Oct 30 13:18:39.833164 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.832609    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-istio-envoy\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh\" (UID: \"8bf396f4-4bbf-46dd-834e-2621e8c1f6cb\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh"
Oct 30 13:18:39.833164 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.832714    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-workload-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh\" (UID: \"8bf396f4-4bbf-46dd-834e-2621e8c1f6cb\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh"
Oct 30 13:18:39.849168 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.848850    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt7sd\" (UniqueName: \"kubernetes.io/projected/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-kube-api-access-gt7sd\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh\" (UID: \"8bf396f4-4bbf-46dd-834e-2621e8c1f6cb\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh"
Oct 30 13:18:39.866883 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.866808    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-istio-token\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh\" (UID: \"8bf396f4-4bbf-46dd-834e-2621e8c1f6cb\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh"
Oct 30 13:18:39.944550 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:39.944511    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn"
Oct 30 13:18:40.139170 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:40.138588    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh"
Oct 30 13:18:40.139506 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:40.139472    5093 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 13:18:40.139697 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:40.139671    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx" podUID="67c19bdd-7c73-46e7-8e2b-58eef9290ce0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 13:18:40.910245 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:40.907266    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx" podUID="ba156eeb-9955-4f9a-857b-2c4d502a5fcc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:40.972156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:40.965233    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv" podUID="d6a44981-104f-4d3e-9937-d184a6ddc482" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:40.997072 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:40.996256    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25" podUID="7805f89f-89d4-4754-b606-c3070f4df215" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:41.075773 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:41.067700    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt" podUID="4ae52c69-0c30-438f-9ac0-c87b6de16061" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:41.075773 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:41.069838    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb" podUID="c4b4e00e-f58d-4a4e-b7db-19f724360db1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:41.153407 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:41.139896    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw" podUID="ad37954e-1350-4f75-a63f-a0c0f29ae416" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:41.170291 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:41.165264    5093 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 13:18:41.170291 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:41.165334    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx" podUID="67c19bdd-7c73-46e7-8e2b-58eef9290ce0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 13:18:41.245245 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:41.240526    5093 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt" podUID="4ae52c69-0c30-438f-9ac0-c87b6de16061" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": context deadline exceeded"
Oct 30 13:18:41.946185 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:41.912306    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx" podUID="ba156eeb-9955-4f9a-857b-2c4d502a5fcc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:42.021739 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:41.964235    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg" podUID="ea079ac1-f502-4b5f-b16a-c91e2de98de8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:42.526156 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:18:42.518979    5093 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.526s"
Oct 30 13:18:42.526156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:42.519097    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx"
Oct 30 13:18:42.646212 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:42.570587    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb" podUID="c4b4e00e-f58d-4a4e-b7db-19f724360db1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Oct 30 13:18:42.646212 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:42.576101    5093 trace.go:236] Trace[198385265]: "Calculate volume metrics of manifests for pod opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-rd5mx" (30-Oct-2026 13:18:41.521) (total time: 1054ms):
Oct 30 13:18:42.646212 ip-10-0-134-31 kubenswrapper[5093]: Trace[198385265]: [1.054537961s] [1.054537961s] END
Oct 30 13:18:42.646212 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:42.589381    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw" podUID="ad37954e-1350-4f75-a63f-a0c0f29ae416" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 13:18:42.646212 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:42.614591    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb"
Oct 30 13:18:42.646212 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:42.614638    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn"]
Oct 30 13:18:42.646212 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:42.614688    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh"]
Oct 30 13:18:42.646212 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:42.614776    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw"
Oct 30 13:18:42.759407 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:42.688454    5093 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 13:18:42.759407 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:42.688553    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx" podUID="67c19bdd-7c73-46e7-8e2b-58eef9290ce0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 13:18:42.862633 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:42.858470    5093 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw" podUID="ad37954e-1350-4f75-a63f-a0c0f29ae416" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:42.970416 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:42.899050    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7hplr" podUID="51cafd88-c303-4185-ac30-aa9f20b1ed4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:42.970416 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:42.937856    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx" podUID="ba156eeb-9955-4f9a-857b-2c4d502a5fcc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:43.125319 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:43.029513    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg" podUID="ea079ac1-f502-4b5f-b16a-c91e2de98de8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Oct 30 13:18:43.125319 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:43.029770    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg"
Oct 30 13:18:43.125319 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:43.063782    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:18:43.125319 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:43.063882    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:18:43.125319 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:43.063923    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:18:43.125319 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:43.081362    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv" podUID="d6a44981-104f-4d3e-9937-d184a6ddc482" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 13:18:43.213213 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:43.184520    5093 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 13:18:43.287573 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:43.287523    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx" podUID="67c19bdd-7c73-46e7-8e2b-58eef9290ce0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 13:18:43.287857 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:43.260639    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:18:43.288095 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:43.288064    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:18:43.296458 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:43.296416    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:18:43.297190 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:43.215784    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt" podUID="4ae52c69-0c30-438f-9ac0-c87b6de16061" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 30 13:18:43.406375 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:43.406273    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25" podUID="7805f89f-89d4-4754-b606-c3070f4df215" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 13:18:43.558668 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:43.558625    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv"
Oct 30 13:18:43.559001 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:43.558982    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt"
Oct 30 13:18:43.574402 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:43.574364    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb" podUID="c4b4e00e-f58d-4a4e-b7db-19f724360db1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Oct 30 13:18:43.597183 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:43.597121    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw" podUID="ad37954e-1350-4f75-a63f-a0c0f29ae416" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:43.950257 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:43.950206    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx" podUID="ba156eeb-9955-4f9a-857b-2c4d502a5fcc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 30 13:18:44.022515 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:44.022164    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt"
Oct 30 13:18:44.078160 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:44.061188    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg" podUID="ea079ac1-f502-4b5f-b16a-c91e2de98de8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:44.107900 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:44.088789    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv" podUID="d6a44981-104f-4d3e-9937-d184a6ddc482" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 13:18:44.198968 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:44.198932    5093 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 13:18:44.204890 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:44.204854    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx" podUID="67c19bdd-7c73-46e7-8e2b-58eef9290ce0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 13:18:44.205167 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:44.202587    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx"
Oct 30 13:18:44.205298 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:44.205281    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh" event={"ID":"8bf396f4-4bbf-46dd-834e-2621e8c1f6cb","Type":"ContainerStarted","Data":"728fd2161210e75fb87b34835f8b1072811f9e8f0f0dbf3b397b671227f11006"}
Oct 30 13:18:44.206404 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:44.206380    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg"
Oct 30 13:18:44.215808 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:44.215770    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn" event={"ID":"874dabda-7e3e-44eb-9768-7ae00d30d22e","Type":"ContainerStarted","Data":"8869023f376be1454e1c66d067e72f466e6e5ffd2a4a98016396e2c98e8fdb5f"}
Oct 30 13:18:44.400986 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:44.400954    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv"
Oct 30 13:18:44.417351 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:44.403419    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw"
Oct 30 13:18:44.417351 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:44.413981    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb"
Oct 30 13:18:44.433156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:44.432155    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25" podUID="7805f89f-89d4-4754-b606-c3070f4df215" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:44.456262 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:44.442818    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25"
Oct 30 13:18:44.519361 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:44.511420    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25"
Oct 30 13:18:45.135799 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:45.135747    5093 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 13:18:45.136480 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:45.136448    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx" podUID="67c19bdd-7c73-46e7-8e2b-58eef9290ce0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 13:18:45.271152 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:45.265703    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh" event={"ID":"8bf396f4-4bbf-46dd-834e-2621e8c1f6cb","Type":"ContainerStarted","Data":"8f6368b1b51273a470c57fba46f70113809bb2796759a5a4e169155ac83d1f12"}
Oct 30 13:18:45.296091 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:45.296047    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn" event={"ID":"874dabda-7e3e-44eb-9768-7ae00d30d22e","Type":"ContainerStarted","Data":"aee13bbffc9f102ba47b22b3311d8dce547ba0fcb4c925664528bb236704942e"}
Oct 30 13:18:45.491288 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:45.491201    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh" podStartSLOduration=7.491121718 podStartE2EDuration="7.491121718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:45.387592103 +0000 UTC m=+944.534859345" watchObservedRunningTime="2026-10-30 13:18:45.491121718 +0000 UTC m=+944.638388960"
Oct 30 13:18:45.960206 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:45.960173    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn"
Oct 30 13:18:45.972447 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:45.972406    5093 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 13:18:45.972730 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:45.972700    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn" podUID="874dabda-7e3e-44eb-9768-7ae00d30d22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 13:18:46.155169 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:46.149381    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh"
Oct 30 13:18:46.186157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:46.176666    5093 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 13:18:46.186157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:46.176754    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh" podUID="8bf396f4-4bbf-46dd-834e-2621e8c1f6cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 13:18:46.800543 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:46.800500    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx"
Oct 30 13:18:46.806583 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:46.805910    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx"
Oct 30 13:18:46.807193 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:46.807143    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx"
Oct 30 13:18:46.893086 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:46.893022    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn" podStartSLOduration=8.892999199 podStartE2EDuration="8.892999199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:45.454508931 +0000 UTC m=+944.601776172" watchObservedRunningTime="2026-10-30 13:18:46.892999199 +0000 UTC m=+946.040266441"
Oct 30 13:18:47.014929 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:47.014891    5093 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 13:18:47.015639 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:47.015196    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn" podUID="874dabda-7e3e-44eb-9768-7ae00d30d22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 13:18:47.182958 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:47.181687    5093 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 13:18:47.182958 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:47.181749    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh" podUID="8bf396f4-4bbf-46dd-834e-2621e8c1f6cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 13:18:47.256155 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:47.252054    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9fdad5-94cb96f84-jj67w"]
Oct 30 13:18:47.308569 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:47.306553    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9fdad5-94cb96f84-jj67w"]
Oct 30 13:18:47.308569 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:47.306704    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9fdad5-94cb96f84-jj67w"
Oct 30 13:18:47.317729 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:47.317696    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b8d2e06-55a6-46ec-940c-6d645814eb8d-maas-api-tls\") pod \"maas-api-e2e-api-a-9fdad5-94cb96f84-jj67w\" (UID: \"9b8d2e06-55a6-46ec-940c-6d645814eb8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9fdad5-94cb96f84-jj67w"
Oct 30 13:18:47.317919 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:47.317848    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b8d2e06-55a6-46ec-940c-6d645814eb8d-metrics-certs\") pod \"maas-api-e2e-api-a-9fdad5-94cb96f84-jj67w\" (UID: \"9b8d2e06-55a6-46ec-940c-6d645814eb8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9fdad5-94cb96f84-jj67w"
Oct 30 13:18:47.318025 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:47.317934    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b8d2e06-55a6-46ec-940c-6d645814eb8d-openshift-service-ca\") pod \"maas-api-e2e-api-a-9fdad5-94cb96f84-jj67w\" (UID: \"9b8d2e06-55a6-46ec-940c-6d645814eb8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9fdad5-94cb96f84-jj67w"
Oct 30 13:18:47.318205 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:47.318180    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdxlt\" (UniqueName: \"kubernetes.io/projected/9b8d2e06-55a6-46ec-940c-6d645814eb8d-kube-api-access-jdxlt\") pod \"maas-api-e2e-api-a-9fdad5-94cb96f84-jj67w\" (UID: \"9b8d2e06-55a6-46ec-940c-6d645814eb8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9fdad5-94cb96f84-jj67w"
Oct 30 13:18:47.323665 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:47.323643    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-9fdad5-serving-cert\""
Oct 30 13:18:47.420856 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:47.420810    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdxlt\" (UniqueName: \"kubernetes.io/projected/9b8d2e06-55a6-46ec-940c-6d645814eb8d-kube-api-access-jdxlt\") pod \"maas-api-e2e-api-a-9fdad5-94cb96f84-jj67w\" (UID: \"9b8d2e06-55a6-46ec-940c-6d645814eb8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9fdad5-94cb96f84-jj67w"
Oct 30 13:18:47.421068 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:47.420937    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b8d2e06-55a6-46ec-940c-6d645814eb8d-maas-api-tls\") pod \"maas-api-e2e-api-a-9fdad5-94cb96f84-jj67w\" (UID: \"9b8d2e06-55a6-46ec-940c-6d645814eb8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9fdad5-94cb96f84-jj67w"
Oct 30 13:18:47.421068 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:47.420977    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b8d2e06-55a6-46ec-940c-6d645814eb8d-metrics-certs\") pod \"maas-api-e2e-api-a-9fdad5-94cb96f84-jj67w\" (UID: \"9b8d2e06-55a6-46ec-940c-6d645814eb8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9fdad5-94cb96f84-jj67w"
Oct 30 13:18:47.421068 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:47.421031    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b8d2e06-55a6-46ec-940c-6d645814eb8d-openshift-service-ca\") pod \"maas-api-e2e-api-a-9fdad5-94cb96f84-jj67w\" (UID: \"9b8d2e06-55a6-46ec-940c-6d645814eb8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9fdad5-94cb96f84-jj67w"
Oct 30 13:18:47.422188 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:47.421888    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b8d2e06-55a6-46ec-940c-6d645814eb8d-openshift-service-ca\") pod \"maas-api-e2e-api-a-9fdad5-94cb96f84-jj67w\" (UID: \"9b8d2e06-55a6-46ec-940c-6d645814eb8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9fdad5-94cb96f84-jj67w"
Oct 30 13:18:47.422503 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:18:47.422479    5093 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-9fdad5-serving-cert: secret "maas-api-e2e-api-a-9fdad5-serving-cert" not found
Oct 30 13:18:47.422603 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:18:47.422541    5093 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b8d2e06-55a6-46ec-940c-6d645814eb8d-maas-api-tls podName:9b8d2e06-55a6-46ec-940c-6d645814eb8d nodeName:}" failed. No retries permitted until 2026-10-30 13:18:47.922521811 +0000 UTC m=+947.069789030 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9b8d2e06-55a6-46ec-940c-6d645814eb8d-maas-api-tls") pod "maas-api-e2e-api-a-9fdad5-94cb96f84-jj67w" (UID: "9b8d2e06-55a6-46ec-940c-6d645814eb8d") : secret "maas-api-e2e-api-a-9fdad5-serving-cert" not found
Oct 30 13:18:47.472776 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:47.460698    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdxlt\" (UniqueName: \"kubernetes.io/projected/9b8d2e06-55a6-46ec-940c-6d645814eb8d-kube-api-access-jdxlt\") pod \"maas-api-e2e-api-a-9fdad5-94cb96f84-jj67w\" (UID: \"9b8d2e06-55a6-46ec-940c-6d645814eb8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9fdad5-94cb96f84-jj67w"
Oct 30 13:18:47.501189 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:47.488823    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b8d2e06-55a6-46ec-940c-6d645814eb8d-metrics-certs\") pod \"maas-api-e2e-api-a-9fdad5-94cb96f84-jj67w\" (UID: \"9b8d2e06-55a6-46ec-940c-6d645814eb8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9fdad5-94cb96f84-jj67w"
Oct 30 13:18:47.950759 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:47.950721    5093 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 13:18:47.951124 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:47.951077    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn" podUID="874dabda-7e3e-44eb-9768-7ae00d30d22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 13:18:47.973181 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:47.972159    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b8d2e06-55a6-46ec-940c-6d645814eb8d-maas-api-tls\") pod \"maas-api-e2e-api-a-9fdad5-94cb96f84-jj67w\" (UID: \"9b8d2e06-55a6-46ec-940c-6d645814eb8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9fdad5-94cb96f84-jj67w"
Oct 30 13:18:47.997824 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:47.997790    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b8d2e06-55a6-46ec-940c-6d645814eb8d-maas-api-tls\") pod \"maas-api-e2e-api-a-9fdad5-94cb96f84-jj67w\" (UID: \"9b8d2e06-55a6-46ec-940c-6d645814eb8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9fdad5-94cb96f84-jj67w"
Oct 30 13:18:48.023765 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:48.023713    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9fdad5-94cb96f84-jj67w"
Oct 30 13:18:48.161153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:48.154582    5093 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 13:18:48.161153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:48.154649    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh" podUID="8bf396f4-4bbf-46dd-834e-2621e8c1f6cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 13:18:48.807155 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:48.806677    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9fdad5-94cb96f84-jj67w"]
Oct 30 13:18:48.820718 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:18:48.820104    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b8d2e06_55a6_46ec_940c_6d645814eb8d.slice/crio-7ca4ea882c93a04bacdea71a16da2025a14b8e26c96cfa061b2270b81e9ecdaa WatchSource:0}: Error finding container 7ca4ea882c93a04bacdea71a16da2025a14b8e26c96cfa061b2270b81e9ecdaa: Status 404 returned error can't find the container with id 7ca4ea882c93a04bacdea71a16da2025a14b8e26c96cfa061b2270b81e9ecdaa
Oct 30 13:18:48.822108 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:48.822080    5093 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 13:18:48.979699 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:48.979657    5093 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 13:18:48.980000 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:48.979965    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn" podUID="874dabda-7e3e-44eb-9768-7ae00d30d22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 13:18:49.160422 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:49.160296    5093 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 13:18:49.160752 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:49.160707    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh" podUID="8bf396f4-4bbf-46dd-834e-2621e8c1f6cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 13:18:49.720227 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:49.719515    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9fdad5-94cb96f84-jj67w" event={"ID":"9b8d2e06-55a6-46ec-940c-6d645814eb8d","Type":"ContainerStarted","Data":"7ca4ea882c93a04bacdea71a16da2025a14b8e26c96cfa061b2270b81e9ecdaa"}
Oct 30 13:18:49.720572 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:49.720534    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9fdad5-94cb96f84-jj67w"
Oct 30 13:18:49.804161 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:49.803092    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9fdad5-94cb96f84-jj67w" podStartSLOduration=2.184878354 podStartE2EDuration="2.803068325s" totalImagesPullingTime="618.189971ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:49.764652888 +0000 UTC m=+948.911920265" watchObservedRunningTime="2026-10-30 13:18:49.803068325 +0000 UTC m=+948.950335568"
Oct 30 13:18:49.960155 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:49.953536    5093 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 13:18:49.960155 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:49.953598    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn" podUID="874dabda-7e3e-44eb-9768-7ae00d30d22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 13:18:50.151214 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:50.147664    5093 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 13:18:50.151214 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:50.147738    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh" podUID="8bf396f4-4bbf-46dd-834e-2621e8c1f6cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 13:18:50.801086 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:50.800936    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9fdad5-94cb96f84-jj67w" event={"ID":"9b8d2e06-55a6-46ec-940c-6d645814eb8d","Type":"ContainerStarted","Data":"bdf3a0db72152a46fc92917ef89a94ea7a491bb8df9e378a96a28930e7262d5d"}
Oct 30 13:18:50.950307 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:50.948729    5093 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 13:18:50.950307 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:50.948798    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn" podUID="874dabda-7e3e-44eb-9768-7ae00d30d22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 13:18:51.144234 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:51.141000    5093 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 13:18:51.144234 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:51.141066    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh" podUID="8bf396f4-4bbf-46dd-834e-2621e8c1f6cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 13:18:51.184317 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:18:51.184272    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:18:52.504711 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:52.504666    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn"
Oct 30 13:18:52.506726 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:52.506697    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn"
Oct 30 13:18:52.507691 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:52.507670    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn"
Oct 30 13:18:52.685229 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:52.681416    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh"
Oct 30 13:18:52.700150 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:52.692956    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh"
Oct 30 13:18:52.708170 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:52.706447    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh"
Oct 30 13:18:56.852346 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:56.852300    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9fdad5-94cb96f84-jj67w"
Oct 30 13:18:57.645306 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:18:57.645256    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw" podUID="ad37954e-1350-4f75-a63f-a0c0f29ae416" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:19:12.482874 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:12.482817    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-6b7b93a3/e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5"]
Oct 30 13:19:12.492767 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:12.491690    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6b7b93a3/e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5"
Oct 30 13:19:12.509936 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:12.509335    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-6b7b93a3/e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5"]
Oct 30 13:19:12.523156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:12.522823    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-6b7b93a3\"/\"kube-root-ca.crt\""
Oct 30 13:19:12.524185 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:12.523770    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-6b7b93a3\"/\"openshift-service-ca.crt\""
Oct 30 13:19:12.524185 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:12.523837    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-6b7b93a3\"/\"default-dockercfg-9d4qr\""
Oct 30 13:19:12.524185 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:12.523907    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-6b7b93a3\"/\"e2e-lifecycle-model-6b7b93a3-kserve-self-signed-certs\""
Oct 30 13:19:12.606330 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:12.606290    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81a50a32-4858-4cfc-94d0-754c8e89f6a4-tmp-dir\") pod \"e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5\" (UID: \"81a50a32-4858-4cfc-94d0-754c8e89f6a4\") " pod="ai-tenant-e2e-mt-6b7b93a3/e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5"
Oct 30 13:19:12.606505 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:12.606356    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnm98\" (UniqueName: \"kubernetes.io/projected/81a50a32-4858-4cfc-94d0-754c8e89f6a4-kube-api-access-bnm98\") pod \"e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5\" (UID: \"81a50a32-4858-4cfc-94d0-754c8e89f6a4\") " pod="ai-tenant-e2e-mt-6b7b93a3/e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5"
Oct 30 13:19:12.606556 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:12.606511    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81a50a32-4858-4cfc-94d0-754c8e89f6a4-tls-certs\") pod \"e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5\" (UID: \"81a50a32-4858-4cfc-94d0-754c8e89f6a4\") " pod="ai-tenant-e2e-mt-6b7b93a3/e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5"
Oct 30 13:19:12.606744 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:12.606728    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81a50a32-4858-4cfc-94d0-754c8e89f6a4-dshm\") pod \"e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5\" (UID: \"81a50a32-4858-4cfc-94d0-754c8e89f6a4\") " pod="ai-tenant-e2e-mt-6b7b93a3/e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5"
Oct 30 13:19:12.606854 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:12.606840    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81a50a32-4858-4cfc-94d0-754c8e89f6a4-home\") pod \"e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5\" (UID: \"81a50a32-4858-4cfc-94d0-754c8e89f6a4\") " pod="ai-tenant-e2e-mt-6b7b93a3/e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5"
Oct 30 13:19:12.606911 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:12.606897    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81a50a32-4858-4cfc-94d0-754c8e89f6a4-model-cache\") pod \"e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5\" (UID: \"81a50a32-4858-4cfc-94d0-754c8e89f6a4\") " pod="ai-tenant-e2e-mt-6b7b93a3/e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5"
Oct 30 13:19:12.708466 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:12.708425    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81a50a32-4858-4cfc-94d0-754c8e89f6a4-dshm\") pod \"e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5\" (UID: \"81a50a32-4858-4cfc-94d0-754c8e89f6a4\") " pod="ai-tenant-e2e-mt-6b7b93a3/e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5"
Oct 30 13:19:12.708663 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:12.708558    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81a50a32-4858-4cfc-94d0-754c8e89f6a4-home\") pod \"e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5\" (UID: \"81a50a32-4858-4cfc-94d0-754c8e89f6a4\") " pod="ai-tenant-e2e-mt-6b7b93a3/e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5"
Oct 30 13:19:12.708663 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:12.708592    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81a50a32-4858-4cfc-94d0-754c8e89f6a4-model-cache\") pod \"e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5\" (UID: \"81a50a32-4858-4cfc-94d0-754c8e89f6a4\") " pod="ai-tenant-e2e-mt-6b7b93a3/e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5"
Oct 30 13:19:12.708663 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:12.708656    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81a50a32-4858-4cfc-94d0-754c8e89f6a4-tmp-dir\") pod \"e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5\" (UID: \"81a50a32-4858-4cfc-94d0-754c8e89f6a4\") " pod="ai-tenant-e2e-mt-6b7b93a3/e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5"
Oct 30 13:19:12.708825 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:12.708722    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnm98\" (UniqueName: \"kubernetes.io/projected/81a50a32-4858-4cfc-94d0-754c8e89f6a4-kube-api-access-bnm98\") pod \"e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5\" (UID: \"81a50a32-4858-4cfc-94d0-754c8e89f6a4\") " pod="ai-tenant-e2e-mt-6b7b93a3/e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5"
Oct 30 13:19:12.708825 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:12.708793    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81a50a32-4858-4cfc-94d0-754c8e89f6a4-tls-certs\") pod \"e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5\" (UID: \"81a50a32-4858-4cfc-94d0-754c8e89f6a4\") " pod="ai-tenant-e2e-mt-6b7b93a3/e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5"
Oct 30 13:19:12.709016 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:12.708992    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81a50a32-4858-4cfc-94d0-754c8e89f6a4-home\") pod \"e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5\" (UID: \"81a50a32-4858-4cfc-94d0-754c8e89f6a4\") " pod="ai-tenant-e2e-mt-6b7b93a3/e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5"
Oct 30 13:19:12.709105 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:12.709058    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81a50a32-4858-4cfc-94d0-754c8e89f6a4-model-cache\") pod \"e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5\" (UID: \"81a50a32-4858-4cfc-94d0-754c8e89f6a4\") " pod="ai-tenant-e2e-mt-6b7b93a3/e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5"
Oct 30 13:19:12.709190 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:12.709116    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81a50a32-4858-4cfc-94d0-754c8e89f6a4-tmp-dir\") pod \"e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5\" (UID: \"81a50a32-4858-4cfc-94d0-754c8e89f6a4\") " pod="ai-tenant-e2e-mt-6b7b93a3/e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5"
Oct 30 13:19:12.710742 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:12.710719    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81a50a32-4858-4cfc-94d0-754c8e89f6a4-dshm\") pod \"e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5\" (UID: \"81a50a32-4858-4cfc-94d0-754c8e89f6a4\") " pod="ai-tenant-e2e-mt-6b7b93a3/e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5"
Oct 30 13:19:12.711288 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:12.711271    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81a50a32-4858-4cfc-94d0-754c8e89f6a4-tls-certs\") pod \"e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5\" (UID: \"81a50a32-4858-4cfc-94d0-754c8e89f6a4\") " pod="ai-tenant-e2e-mt-6b7b93a3/e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5"
Oct 30 13:19:12.729259 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:12.729217    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnm98\" (UniqueName: \"kubernetes.io/projected/81a50a32-4858-4cfc-94d0-754c8e89f6a4-kube-api-access-bnm98\") pod \"e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5\" (UID: \"81a50a32-4858-4cfc-94d0-754c8e89f6a4\") " pod="ai-tenant-e2e-mt-6b7b93a3/e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5"
Oct 30 13:19:12.821855 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:12.821558    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6b7b93a3/e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5"
Oct 30 13:19:13.162014 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:13.161982    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-6b7b93a3/e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5"]
Oct 30 13:19:13.164253 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:19:13.164113    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81a50a32_4858_4cfc_94d0_754c8e89f6a4.slice/crio-178f355d8328bf9718adae7eafc44530652cad05d55d99eba60edbcfab522238 WatchSource:0}: Error finding container 178f355d8328bf9718adae7eafc44530652cad05d55d99eba60edbcfab522238: Status 404 returned error can't find the container with id 178f355d8328bf9718adae7eafc44530652cad05d55d99eba60edbcfab522238
Oct 30 13:19:14.122512 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:14.122470    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6b7b93a3/e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5" event={"ID":"81a50a32-4858-4cfc-94d0-754c8e89f6a4","Type":"ContainerStarted","Data":"929ac8585ac4e7d4661e9d2ea4c9eff0cd71ec823f2fbf290a82393d1c7dfcd5"}
Oct 30 13:19:14.123082 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:14.122528    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6b7b93a3/e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5" event={"ID":"81a50a32-4858-4cfc-94d0-754c8e89f6a4","Type":"ContainerStarted","Data":"178f355d8328bf9718adae7eafc44530652cad05d55d99eba60edbcfab522238"}
Oct 30 13:19:14.197154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:14.195072    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-6b7b93a3/e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5" podStartSLOduration=2.195053748 podStartE2EDuration="2.195053748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:14.181279078 +0000 UTC m=+973.328546320" watchObservedRunningTime="2026-10-30 13:19:14.195053748 +0000 UTC m=+973.342320989"
Oct 30 13:19:14.829184 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:14.822237    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-6b7b93a3/e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5"
Oct 30 13:19:21.173576 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:19:21.173529    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:19:22.822207 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:22.822173    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-6b7b93a3/e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5"
Oct 30 13:19:22.835694 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:22.835660    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-6b7b93a3/e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5"
Oct 30 13:19:23.462522 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:23.462483    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-6b7b93a3/e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5"
Oct 30 13:19:25.471646 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:25.471553    5093 generic.go:356] "Generic (PLEG): container finished" podID="528a3a61-adc9-45fd-aefd-1cc67d8cf716" containerID="c7944a7e350c9ad5a27bc5f7a09cabd57d9ad46fdb7ec8104d2a6c7186d462d4" exitCode=0
Oct 30 13:19:25.471646 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:25.471626    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf" event={"ID":"528a3a61-adc9-45fd-aefd-1cc67d8cf716","Type":"ContainerDied","Data":"c7944a7e350c9ad5a27bc5f7a09cabd57d9ad46fdb7ec8104d2a6c7186d462d4"}
Oct 30 13:19:25.472284 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:25.472232    5093 scope.go:122] "RemoveContainer" containerID="c7944a7e350c9ad5a27bc5f7a09cabd57d9ad46fdb7ec8104d2a6c7186d462d4"
Oct 30 13:19:25.473946 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:25.473905    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:19:25.474053 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:25.473983    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:19:25.474053 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:25.474038    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:19:26.487296 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:26.487256    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf" event={"ID":"528a3a61-adc9-45fd-aefd-1cc67d8cf716","Type":"ContainerStarted","Data":"58e77bdcd8cfff405a7118e261acf4340e27edd9f57e3fa81bfb663fa4a95ac7"}
Oct 30 13:19:27.214415 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:27.214378    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf"
Oct 30 13:19:27.214629 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:27.214602    5093 patch_prober.go:29] interesting pod/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 13:19:27.214713 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:27.214642    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf" podUID="528a3a61-adc9-45fd-aefd-1cc67d8cf716" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 13:19:28.214630 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:28.214597    5093 patch_prober.go:29] interesting pod/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 13:19:28.215045 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:28.214660    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf" podUID="528a3a61-adc9-45fd-aefd-1cc67d8cf716" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 13:19:29.215193 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:29.215158    5093 patch_prober.go:29] interesting pod/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 13:19:29.215624 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:29.215215    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf" podUID="528a3a61-adc9-45fd-aefd-1cc67d8cf716" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 13:19:30.214731 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:30.214691    5093 patch_prober.go:29] interesting pod/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 13:19:30.214918 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:30.214761    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf" podUID="528a3a61-adc9-45fd-aefd-1cc67d8cf716" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 13:19:31.215172 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:31.215140    5093 patch_prober.go:29] interesting pod/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 13:19:31.215663 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:31.215636    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf" podUID="528a3a61-adc9-45fd-aefd-1cc67d8cf716" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 13:19:31.278412 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:31.277434    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2ed395-56fd58558-jwsg9"]
Oct 30 13:19:31.283160 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:31.282170    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2ed395-56fd58558-jwsg9"]
Oct 30 13:19:31.283160 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:31.282292    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ed395-56fd58558-jwsg9"
Oct 30 13:19:31.303237 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:31.303208    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-2ed395-serving-cert\""
Oct 30 13:19:31.405046 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:31.405003    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t9b6\" (UniqueName: \"kubernetes.io/projected/99d6ebaa-4374-4502-90db-59d12b6836c0-kube-api-access-9t9b6\") pod \"maas-api-e2e-api-b-2ed395-56fd58558-jwsg9\" (UID: \"99d6ebaa-4374-4502-90db-59d12b6836c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ed395-56fd58558-jwsg9"
Oct 30 13:19:31.405240 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:31.405065    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99d6ebaa-4374-4502-90db-59d12b6836c0-maas-api-tls\") pod \"maas-api-e2e-api-b-2ed395-56fd58558-jwsg9\" (UID: \"99d6ebaa-4374-4502-90db-59d12b6836c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ed395-56fd58558-jwsg9"
Oct 30 13:19:31.405325 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:31.405281    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99d6ebaa-4374-4502-90db-59d12b6836c0-openshift-service-ca\") pod \"maas-api-e2e-api-b-2ed395-56fd58558-jwsg9\" (UID: \"99d6ebaa-4374-4502-90db-59d12b6836c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ed395-56fd58558-jwsg9"
Oct 30 13:19:31.405368 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:31.405347    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99d6ebaa-4374-4502-90db-59d12b6836c0-metrics-certs\") pod \"maas-api-e2e-api-b-2ed395-56fd58558-jwsg9\" (UID: \"99d6ebaa-4374-4502-90db-59d12b6836c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ed395-56fd58558-jwsg9"
Oct 30 13:19:31.506657 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:31.506561    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99d6ebaa-4374-4502-90db-59d12b6836c0-openshift-service-ca\") pod \"maas-api-e2e-api-b-2ed395-56fd58558-jwsg9\" (UID: \"99d6ebaa-4374-4502-90db-59d12b6836c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ed395-56fd58558-jwsg9"
Oct 30 13:19:31.506657 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:31.506632    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99d6ebaa-4374-4502-90db-59d12b6836c0-metrics-certs\") pod \"maas-api-e2e-api-b-2ed395-56fd58558-jwsg9\" (UID: \"99d6ebaa-4374-4502-90db-59d12b6836c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ed395-56fd58558-jwsg9"
Oct 30 13:19:31.506914 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:31.506780    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t9b6\" (UniqueName: \"kubernetes.io/projected/99d6ebaa-4374-4502-90db-59d12b6836c0-kube-api-access-9t9b6\") pod \"maas-api-e2e-api-b-2ed395-56fd58558-jwsg9\" (UID: \"99d6ebaa-4374-4502-90db-59d12b6836c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ed395-56fd58558-jwsg9"
Oct 30 13:19:31.506914 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:31.506810    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99d6ebaa-4374-4502-90db-59d12b6836c0-maas-api-tls\") pod \"maas-api-e2e-api-b-2ed395-56fd58558-jwsg9\" (UID: \"99d6ebaa-4374-4502-90db-59d12b6836c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ed395-56fd58558-jwsg9"
Oct 30 13:19:31.508120 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:31.508087    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99d6ebaa-4374-4502-90db-59d12b6836c0-openshift-service-ca\") pod \"maas-api-e2e-api-b-2ed395-56fd58558-jwsg9\" (UID: \"99d6ebaa-4374-4502-90db-59d12b6836c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ed395-56fd58558-jwsg9"
Oct 30 13:19:31.509679 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:31.509653    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99d6ebaa-4374-4502-90db-59d12b6836c0-maas-api-tls\") pod \"maas-api-e2e-api-b-2ed395-56fd58558-jwsg9\" (UID: \"99d6ebaa-4374-4502-90db-59d12b6836c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ed395-56fd58558-jwsg9"
Oct 30 13:19:31.509799 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:31.509773    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99d6ebaa-4374-4502-90db-59d12b6836c0-metrics-certs\") pod \"maas-api-e2e-api-b-2ed395-56fd58558-jwsg9\" (UID: \"99d6ebaa-4374-4502-90db-59d12b6836c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ed395-56fd58558-jwsg9"
Oct 30 13:19:31.545911 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:31.545871    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t9b6\" (UniqueName: \"kubernetes.io/projected/99d6ebaa-4374-4502-90db-59d12b6836c0-kube-api-access-9t9b6\") pod \"maas-api-e2e-api-b-2ed395-56fd58558-jwsg9\" (UID: \"99d6ebaa-4374-4502-90db-59d12b6836c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ed395-56fd58558-jwsg9"
Oct 30 13:19:31.605897 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:31.605864    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ed395-56fd58558-jwsg9"
Oct 30 13:19:31.944738 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:31.944704    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2ed395-56fd58558-jwsg9"]
Oct 30 13:19:31.949183 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:19:31.949118    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99d6ebaa_4374_4502_90db_59d12b6836c0.slice/crio-292a6da663df33b2ae2b771fbe8106c8f17c63a2e95b1b60d39f4d4073f9a2c8 WatchSource:0}: Error finding container 292a6da663df33b2ae2b771fbe8106c8f17c63a2e95b1b60d39f4d4073f9a2c8: Status 404 returned error can't find the container with id 292a6da663df33b2ae2b771fbe8106c8f17c63a2e95b1b60d39f4d4073f9a2c8
Oct 30 13:19:32.219212 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:32.219114    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf"
Oct 30 13:19:32.219638 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:32.219395    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf"
Oct 30 13:19:32.220120 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:32.220099    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf"
Oct 30 13:19:32.550122 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:32.549991    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ed395-56fd58558-jwsg9" event={"ID":"99d6ebaa-4374-4502-90db-59d12b6836c0","Type":"ContainerStarted","Data":"34c3d488e0b5adf17b057ab5282614eb479fdd80b7f42be827f433c0dfa3b31c"}
Oct 30 13:19:32.550122 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:32.550072    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ed395-56fd58558-jwsg9" event={"ID":"99d6ebaa-4374-4502-90db-59d12b6836c0","Type":"ContainerStarted","Data":"292a6da663df33b2ae2b771fbe8106c8f17c63a2e95b1b60d39f4d4073f9a2c8"}
Oct 30 13:19:32.550500 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:32.550483    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ed395-56fd58558-jwsg9"
Oct 30 13:19:32.612153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:32.601211    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ed395-56fd58558-jwsg9" podStartSLOduration=1.227447299 podStartE2EDuration="1.601188568s" totalImagesPullingTime="373.741269ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:32.596483072 +0000 UTC m=+991.743750315" watchObservedRunningTime="2026-10-30 13:19:32.601188568 +0000 UTC m=+991.748455811"
Oct 30 13:19:38.583214 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:19:38.583175    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ed395-56fd58558-jwsg9"
Oct 30 13:19:51.167799 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:19:51.167751    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:20:21.177593 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:20:21.177545    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:20:22.175360 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:22.175327    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7680-1de067"]
Oct 30 13:20:22.179844 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:22.179819    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7680-1de067"
Oct 30 13:20:22.194566 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:22.194537    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7680-1de067"]
Oct 30 13:20:22.202723 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:22.202679    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4bkkd\""
Oct 30 13:20:22.299295 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:22.299253    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtmjs\" (UniqueName: \"kubernetes.io/projected/f603bae4-fde5-44a5-8f2f-fbff092375b1-kube-api-access-jtmjs\") pod \"test-curl-7680-1de067\" (UID: \"f603bae4-fde5-44a5-8f2f-fbff092375b1\") " pod="opendatahub/test-curl-7680-1de067"
Oct 30 13:20:22.400822 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:22.400784    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtmjs\" (UniqueName: \"kubernetes.io/projected/f603bae4-fde5-44a5-8f2f-fbff092375b1-kube-api-access-jtmjs\") pod \"test-curl-7680-1de067\" (UID: \"f603bae4-fde5-44a5-8f2f-fbff092375b1\") " pod="opendatahub/test-curl-7680-1de067"
Oct 30 13:20:22.423226 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:22.423189    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtmjs\" (UniqueName: \"kubernetes.io/projected/f603bae4-fde5-44a5-8f2f-fbff092375b1-kube-api-access-jtmjs\") pod \"test-curl-7680-1de067\" (UID: \"f603bae4-fde5-44a5-8f2f-fbff092375b1\") " pod="opendatahub/test-curl-7680-1de067"
Oct 30 13:20:22.495322 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:22.495224    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7680-1de067"
Oct 30 13:20:22.805992 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:22.805958    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7680-1de067"]
Oct 30 13:20:23.074195 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:23.074060    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7680-1de067" event={"ID":"f603bae4-fde5-44a5-8f2f-fbff092375b1","Type":"ContainerStarted","Data":"c7f6f5b604df50613c7bfc1885eb8a577206aef17d4f80f0d1246362dc84f0d4"}
Oct 30 13:20:24.084512 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:24.084473    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7680-1de067" event={"ID":"f603bae4-fde5-44a5-8f2f-fbff092375b1","Type":"ContainerStarted","Data":"cbd53ac945be5fb250f0e439cd3e2ceba7c34c35ccdc1a253935c56263e8f1a1"}
Oct 30 13:20:24.137776 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:24.137717    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7680-1de067" podStartSLOduration=1.591928465 podStartE2EDuration="2.137698287s" totalImagesPullingTime="545.769822ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:24.135123511 +0000 UTC m=+1043.282390753" watchObservedRunningTime="2026-10-30 13:20:24.137698287 +0000 UTC m=+1043.284965530"
Oct 30 13:20:25.137374 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:25.137340    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7680-1de067"]
Oct 30 13:20:25.143270 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:25.143225    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7680-1de067"]
Oct 30 13:20:26.100639 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:26.100598    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7680-1de067" podUID="f603bae4-fde5-44a5-8f2f-fbff092375b1" containerName="test-curl-7680-1de067" containerID="cri-o://cbd53ac945be5fb250f0e439cd3e2ceba7c34c35ccdc1a253935c56263e8f1a1" gracePeriod=2
Oct 30 13:20:26.115205 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:20:26.115164    5093 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7680-1de067\" is forbidden: User \"system:node:ip-10-0-134-31.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-31.ec2.internal' and this object" podUID="f603bae4-fde5-44a5-8f2f-fbff092375b1" pod="opendatahub/test-curl-7680-1de067"
Oct 30 13:20:26.221878 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:26.221836    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9fdad5-94cb96f84-jj67w"]
Oct 30 13:20:26.222479 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:26.222072    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9fdad5-94cb96f84-jj67w" podUID="9b8d2e06-55a6-46ec-940c-6d645814eb8d" containerName="maas-api" containerID="cri-o://bdf3a0db72152a46fc92917ef89a94ea7a491bb8df9e378a96a28930e7262d5d" gracePeriod=30
Oct 30 13:20:26.237093 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:20:26.236476    5093 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7680-1de067\" is forbidden: User \"system:node:ip-10-0-134-31.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-31.ec2.internal' and this object" podUID="f603bae4-fde5-44a5-8f2f-fbff092375b1" pod="opendatahub/test-curl-7680-1de067"
Oct 30 13:20:26.619249 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:26.619217    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9fdad5-94cb96f84-jj67w"
Oct 30 13:20:26.634115 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:20:26.634073    5093 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7680-1de067\" is forbidden: User \"system:node:ip-10-0-134-31.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-31.ec2.internal' and this object" podUID="f603bae4-fde5-44a5-8f2f-fbff092375b1" pod="opendatahub/test-curl-7680-1de067"
Oct 30 13:20:26.795040 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:26.795001    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b8d2e06-55a6-46ec-940c-6d645814eb8d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b8d2e06-55a6-46ec-940c-6d645814eb8d-metrics-certs\") pod \"9b8d2e06-55a6-46ec-940c-6d645814eb8d\" (UID: \"9b8d2e06-55a6-46ec-940c-6d645814eb8d\") "
Oct 30 13:20:26.813113 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:26.795162    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b8d2e06-55a6-46ec-940c-6d645814eb8d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b8d2e06-55a6-46ec-940c-6d645814eb8d-openshift-service-ca\") pod \"9b8d2e06-55a6-46ec-940c-6d645814eb8d\" (UID: \"9b8d2e06-55a6-46ec-940c-6d645814eb8d\") "
Oct 30 13:20:26.813356 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:26.813295    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b8d2e06-55a6-46ec-940c-6d645814eb8d-kube-api-access-jdxlt\" (UniqueName: \"kubernetes.io/projected/9b8d2e06-55a6-46ec-940c-6d645814eb8d-kube-api-access-jdxlt\") pod \"9b8d2e06-55a6-46ec-940c-6d645814eb8d\" (UID: \"9b8d2e06-55a6-46ec-940c-6d645814eb8d\") "
Oct 30 13:20:26.813422 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:26.813357    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b8d2e06-55a6-46ec-940c-6d645814eb8d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b8d2e06-55a6-46ec-940c-6d645814eb8d-maas-api-tls\") pod \"9b8d2e06-55a6-46ec-940c-6d645814eb8d\" (UID: \"9b8d2e06-55a6-46ec-940c-6d645814eb8d\") "
Oct 30 13:20:26.813871 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:26.795921    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b8d2e06-55a6-46ec-940c-6d645814eb8d-openshift-service-ca" pod "9b8d2e06-55a6-46ec-940c-6d645814eb8d" (UID: "9b8d2e06-55a6-46ec-940c-6d645814eb8d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:20:26.813871 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:26.810621    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b8d2e06-55a6-46ec-940c-6d645814eb8d-metrics-certs" pod "9b8d2e06-55a6-46ec-940c-6d645814eb8d" (UID: "9b8d2e06-55a6-46ec-940c-6d645814eb8d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:20:26.814706 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:26.814672    5093 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b8d2e06-55a6-46ec-940c-6d645814eb8d-openshift-service-ca\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:20:26.814817 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:26.814712    5093 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b8d2e06-55a6-46ec-940c-6d645814eb8d-metrics-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:20:26.862874 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:26.862834    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b8d2e06-55a6-46ec-940c-6d645814eb8d-kube-api-access-jdxlt" pod "9b8d2e06-55a6-46ec-940c-6d645814eb8d" (UID: "9b8d2e06-55a6-46ec-940c-6d645814eb8d"). InnerVolumeSpecName "kube-api-access-jdxlt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:20:26.874512 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:26.874473    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b8d2e06-55a6-46ec-940c-6d645814eb8d-maas-api-tls" pod "9b8d2e06-55a6-46ec-940c-6d645814eb8d" (UID: "9b8d2e06-55a6-46ec-940c-6d645814eb8d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:20:26.916092 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:26.916032    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdxlt\" (UniqueName: \"kubernetes.io/projected/9b8d2e06-55a6-46ec-940c-6d645814eb8d-kube-api-access-jdxlt\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:20:26.916092 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:26.916094    5093 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b8d2e06-55a6-46ec-940c-6d645814eb8d-maas-api-tls\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:20:27.112276 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:27.112108    5093 generic.go:356] "Generic (PLEG): container finished" podID="9b8d2e06-55a6-46ec-940c-6d645814eb8d" containerID="bdf3a0db72152a46fc92917ef89a94ea7a491bb8df9e378a96a28930e7262d5d" exitCode=0
Oct 30 13:20:27.112470 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:27.112187    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9fdad5-94cb96f84-jj67w"
Oct 30 13:20:27.112531 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:27.112183    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9fdad5-94cb96f84-jj67w" event={"ID":"9b8d2e06-55a6-46ec-940c-6d645814eb8d","Type":"ContainerDied","Data":"bdf3a0db72152a46fc92917ef89a94ea7a491bb8df9e378a96a28930e7262d5d"}
Oct 30 13:20:27.112581 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:27.112525    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9fdad5-94cb96f84-jj67w" event={"ID":"9b8d2e06-55a6-46ec-940c-6d645814eb8d","Type":"ContainerDied","Data":"7ca4ea882c93a04bacdea71a16da2025a14b8e26c96cfa061b2270b81e9ecdaa"}
Oct 30 13:20:27.112581 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:27.112561    5093 scope.go:122] "RemoveContainer" containerID="bdf3a0db72152a46fc92917ef89a94ea7a491bb8df9e378a96a28930e7262d5d"
Oct 30 13:20:27.127179 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:27.127149    5093 scope.go:122] "RemoveContainer" containerID="bdf3a0db72152a46fc92917ef89a94ea7a491bb8df9e378a96a28930e7262d5d"
Oct 30 13:20:27.127674 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:20:27.127643    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdf3a0db72152a46fc92917ef89a94ea7a491bb8df9e378a96a28930e7262d5d\": container with ID starting with bdf3a0db72152a46fc92917ef89a94ea7a491bb8df9e378a96a28930e7262d5d not found: ID does not exist" containerID="bdf3a0db72152a46fc92917ef89a94ea7a491bb8df9e378a96a28930e7262d5d"
Oct 30 13:20:27.127800 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:27.127695    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdf3a0db72152a46fc92917ef89a94ea7a491bb8df9e378a96a28930e7262d5d"} err="failed to get container status \"bdf3a0db72152a46fc92917ef89a94ea7a491bb8df9e378a96a28930e7262d5d\": rpc error: code = NotFound desc = could not find container \"bdf3a0db72152a46fc92917ef89a94ea7a491bb8df9e378a96a28930e7262d5d\": container with ID starting with bdf3a0db72152a46fc92917ef89a94ea7a491bb8df9e378a96a28930e7262d5d not found: ID does not exist"
Oct 30 13:20:27.172592 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:27.172553    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9fdad5-94cb96f84-jj67w"]
Oct 30 13:20:27.184962 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:27.184921    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9fdad5-94cb96f84-jj67w"]
Oct 30 13:20:28.126239 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:28.126203    5093 generic.go:356] "Generic (PLEG): container finished" podID="f603bae4-fde5-44a5-8f2f-fbff092375b1" containerID="cbd53ac945be5fb250f0e439cd3e2ceba7c34c35ccdc1a253935c56263e8f1a1" exitCode=137
Oct 30 13:20:28.335653 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:28.335628    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7680-1de067"
Oct 30 13:20:28.440922 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:28.440882    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f603bae4-fde5-44a5-8f2f-fbff092375b1-kube-api-access-jtmjs\" (UniqueName: \"kubernetes.io/projected/f603bae4-fde5-44a5-8f2f-fbff092375b1-kube-api-access-jtmjs\") pod \"f603bae4-fde5-44a5-8f2f-fbff092375b1\" (UID: \"f603bae4-fde5-44a5-8f2f-fbff092375b1\") "
Oct 30 13:20:28.443302 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:28.443270    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f603bae4-fde5-44a5-8f2f-fbff092375b1-kube-api-access-jtmjs" pod "f603bae4-fde5-44a5-8f2f-fbff092375b1" (UID: "f603bae4-fde5-44a5-8f2f-fbff092375b1"). InnerVolumeSpecName "kube-api-access-jtmjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:20:28.543114 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:28.543075    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtmjs\" (UniqueName: \"kubernetes.io/projected/f603bae4-fde5-44a5-8f2f-fbff092375b1-kube-api-access-jtmjs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:20:29.002572 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:29.002535    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b8d2e06-55a6-46ec-940c-6d645814eb8d" path="/var/lib/kubelet/pods/9b8d2e06-55a6-46ec-940c-6d645814eb8d/volumes"
Oct 30 13:20:29.003171 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:29.003118    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f603bae4-fde5-44a5-8f2f-fbff092375b1" path="/var/lib/kubelet/pods/f603bae4-fde5-44a5-8f2f-fbff092375b1/volumes"
Oct 30 13:20:29.158829 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:29.158794    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7680-1de067"
Oct 30 13:20:29.159362 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:29.158800    5093 scope.go:122] "RemoveContainer" containerID="cbd53ac945be5fb250f0e439cd3e2ceba7c34c35ccdc1a253935c56263e8f1a1"
Oct 30 13:20:32.390813 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:32.390780    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb"]
Oct 30 13:20:32.391671 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:32.391647    5093 cpu_manager.go:414] "containerMap: removing container" podUID="9b8d2e06-55a6-46ec-940c-6d645814eb8d" containerName="maas-api"
Oct 30 13:20:32.391671 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:32.391670    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b8d2e06-55a6-46ec-940c-6d645814eb8d" containerName="maas-api"
Oct 30 13:20:32.391824 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:32.391725    5093 cpu_manager.go:414] "containerMap: removing container" podUID="f603bae4-fde5-44a5-8f2f-fbff092375b1" containerName="test-curl-7680-1de067"
Oct 30 13:20:32.391824 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:32.391734    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f603bae4-fde5-44a5-8f2f-fbff092375b1" containerName="test-curl-7680-1de067"
Oct 30 13:20:32.391958 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:32.391943    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b8d2e06-55a6-46ec-940c-6d645814eb8d" containerName="maas-api"
Oct 30 13:20:32.392016 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:32.391971    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="f603bae4-fde5-44a5-8f2f-fbff092375b1" containerName="test-curl-7680-1de067"
Oct 30 13:20:32.397020 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:32.396992    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb"
Oct 30 13:20:32.433650 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:32.433617    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-8ac28e-openshift-default-dockercfg-d86tl\""
Oct 30 13:20:32.451271 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:32.451241    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb"]
Oct 30 13:20:32.516823 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:32.516790    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0b3e685-671d-4227-bd45-f3a614caf44d-istio-podinfo\") pod \"e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb\" (UID: \"c0b3e685-671d-4227-bd45-f3a614caf44d\") " pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb"
Oct 30 13:20:32.517019 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:32.516995    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0b3e685-671d-4227-bd45-f3a614caf44d-credential-socket\") pod \"e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb\" (UID: \"c0b3e685-671d-4227-bd45-f3a614caf44d\") " pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb"
Oct 30 13:20:32.517073 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:32.517049    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0b3e685-671d-4227-bd45-f3a614caf44d-workload-certs\") pod \"e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb\" (UID: \"c0b3e685-671d-4227-bd45-f3a614caf44d\") " pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb"
Oct 30 13:20:32.517172 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:32.517120    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0b3e685-671d-4227-bd45-f3a614caf44d-istio-token\") pod \"e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb\" (UID: \"c0b3e685-671d-4227-bd45-f3a614caf44d\") " pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb"
Oct 30 13:20:32.517232 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:32.517205    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0b3e685-671d-4227-bd45-f3a614caf44d-workload-socket\") pod \"e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb\" (UID: \"c0b3e685-671d-4227-bd45-f3a614caf44d\") " pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb"
Oct 30 13:20:32.517344 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:32.517321    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhqn4\" (UniqueName: \"kubernetes.io/projected/c0b3e685-671d-4227-bd45-f3a614caf44d-kube-api-access-zhqn4\") pod \"e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb\" (UID: \"c0b3e685-671d-4227-bd45-f3a614caf44d\") " pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb"
Oct 30 13:20:32.517429 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:32.517409    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0b3e685-671d-4227-bd45-f3a614caf44d-istio-envoy\") pod \"e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb\" (UID: \"c0b3e685-671d-4227-bd45-f3a614caf44d\") " pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb"
Oct 30 13:20:32.517490 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:32.517471    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0b3e685-671d-4227-bd45-f3a614caf44d-istiod-ca-cert\") pod \"e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb\" (UID: \"c0b3e685-671d-4227-bd45-f3a614caf44d\") " pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb"
Oct 30 13:20:32.517573 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:32.517558    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0b3e685-671d-4227-bd45-f3a614caf44d-istio-data\") pod \"e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb\" (UID: \"c0b3e685-671d-4227-bd45-f3a614caf44d\") " pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb"
Oct 30 13:20:32.619520 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:32.619477    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0b3e685-671d-4227-bd45-f3a614caf44d-credential-socket\") pod \"e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb\" (UID: \"c0b3e685-671d-4227-bd45-f3a614caf44d\") " pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb"
Oct 30 13:20:32.619724 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:32.619534    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0b3e685-671d-4227-bd45-f3a614caf44d-workload-certs\") pod \"e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb\" (UID: \"c0b3e685-671d-4227-bd45-f3a614caf44d\") " pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb"
Oct 30 13:20:32.619724 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:32.619584    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0b3e685-671d-4227-bd45-f3a614caf44d-istio-token\") pod \"e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb\" (UID: \"c0b3e685-671d-4227-bd45-f3a614caf44d\") " pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb"
Oct 30 13:20:32.619724 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:32.619625    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0b3e685-671d-4227-bd45-f3a614caf44d-workload-socket\") pod \"e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb\" (UID: \"c0b3e685-671d-4227-bd45-f3a614caf44d\") " pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb"
Oct 30 13:20:32.619724 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:32.619702    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhqn4\" (UniqueName: \"kubernetes.io/projected/c0b3e685-671d-4227-bd45-f3a614caf44d-kube-api-access-zhqn4\") pod \"e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb\" (UID: \"c0b3e685-671d-4227-bd45-f3a614caf44d\") " pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb"
Oct 30 13:20:32.619955 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:32.619745    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0b3e685-671d-4227-bd45-f3a614caf44d-istio-envoy\") pod \"e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb\" (UID: \"c0b3e685-671d-4227-bd45-f3a614caf44d\") " pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb"
Oct 30 13:20:32.619955 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:32.619799    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0b3e685-671d-4227-bd45-f3a614caf44d-istiod-ca-cert\") pod \"e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb\" (UID: \"c0b3e685-671d-4227-bd45-f3a614caf44d\") " pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb"
Oct 30 13:20:32.619955 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:32.619885    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0b3e685-671d-4227-bd45-f3a614caf44d-istio-data\") pod \"e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb\" (UID: \"c0b3e685-671d-4227-bd45-f3a614caf44d\") " pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb"
Oct 30 13:20:32.619955 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:32.619930    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0b3e685-671d-4227-bd45-f3a614caf44d-credential-socket\") pod \"e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb\" (UID: \"c0b3e685-671d-4227-bd45-f3a614caf44d\") " pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb"
Oct 30 13:20:32.619955 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:32.619942    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0b3e685-671d-4227-bd45-f3a614caf44d-istio-podinfo\") pod \"e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb\" (UID: \"c0b3e685-671d-4227-bd45-f3a614caf44d\") " pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb"
Oct 30 13:20:32.620267 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:32.620239    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0b3e685-671d-4227-bd45-f3a614caf44d-workload-certs\") pod \"e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb\" (UID: \"c0b3e685-671d-4227-bd45-f3a614caf44d\") " pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb"
Oct 30 13:20:32.620555 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:32.620530    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0b3e685-671d-4227-bd45-f3a614caf44d-workload-socket\") pod \"e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb\" (UID: \"c0b3e685-671d-4227-bd45-f3a614caf44d\") " pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb"
Oct 30 13:20:32.620952 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:32.620877    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0b3e685-671d-4227-bd45-f3a614caf44d-istiod-ca-cert\") pod \"e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb\" (UID: \"c0b3e685-671d-4227-bd45-f3a614caf44d\") " pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb"
Oct 30 13:20:32.621229 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:32.621202    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0b3e685-671d-4227-bd45-f3a614caf44d-istio-data\") pod \"e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb\" (UID: \"c0b3e685-671d-4227-bd45-f3a614caf44d\") " pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb"
Oct 30 13:20:32.622653 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:32.622633    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0b3e685-671d-4227-bd45-f3a614caf44d-istio-envoy\") pod \"e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb\" (UID: \"c0b3e685-671d-4227-bd45-f3a614caf44d\") " pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb"
Oct 30 13:20:32.623275 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:32.623256    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0b3e685-671d-4227-bd45-f3a614caf44d-istio-podinfo\") pod \"e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb\" (UID: \"c0b3e685-671d-4227-bd45-f3a614caf44d\") " pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb"
Oct 30 13:20:32.679297 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:32.667850    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhqn4\" (UniqueName: \"kubernetes.io/projected/c0b3e685-671d-4227-bd45-f3a614caf44d-kube-api-access-zhqn4\") pod \"e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb\" (UID: \"c0b3e685-671d-4227-bd45-f3a614caf44d\") " pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb"
Oct 30 13:20:32.680101 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:32.680029    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0b3e685-671d-4227-bd45-f3a614caf44d-istio-token\") pod \"e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb\" (UID: \"c0b3e685-671d-4227-bd45-f3a614caf44d\") " pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb"
Oct 30 13:20:32.720343 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:32.719747    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb"
Oct 30 13:20:33.090405 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:33.090365    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb"]
Oct 30 13:20:33.092053 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:20:33.092013    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0b3e685_671d_4227_bd45_f3a614caf44d.slice/crio-b445cdc7ace3ddff0e25b2a56145532416e51cef4d495435c89f19f724aabecf WatchSource:0}: Error finding container b445cdc7ace3ddff0e25b2a56145532416e51cef4d495435c89f19f724aabecf: Status 404 returned error can't find the container with id b445cdc7ace3ddff0e25b2a56145532416e51cef4d495435c89f19f724aabecf
Oct 30 13:20:33.094962 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:33.094922    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:20:33.095086 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:33.095024    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:20:33.095086 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:33.095074    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:20:33.197240 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:33.197199    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb" event={"ID":"c0b3e685-671d-4227-bd45-f3a614caf44d","Type":"ContainerStarted","Data":"941f9890e9b596c68bee2804faf8cf39e419581aac827f00c4f546dd06362a9b"}
Oct 30 13:20:33.197414 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:33.197252    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb" event={"ID":"c0b3e685-671d-4227-bd45-f3a614caf44d","Type":"ContainerStarted","Data":"b445cdc7ace3ddff0e25b2a56145532416e51cef4d495435c89f19f724aabecf"}
Oct 30 13:20:33.244574 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:33.244478    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb" podStartSLOduration=1.244462033 podStartE2EDuration="1.244462033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:33.242367613 +0000 UTC m=+1052.389634855" watchObservedRunningTime="2026-10-30 13:20:33.244462033 +0000 UTC m=+1052.391729273"
Oct 30 13:20:33.719843 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:33.719808    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb"
Oct 30 13:20:34.726556 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:34.726527    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb"
Oct 30 13:20:35.211718 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:35.211685    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb"
Oct 30 13:20:35.212597 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:35.212577    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb"
Oct 30 13:20:39.129524 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:39.127228    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3c2202-98944d8f9-vxvvs"]
Oct 30 13:20:39.160108 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:39.159674    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3c2202-98944d8f9-vxvvs"]
Oct 30 13:20:39.160108 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:39.159814    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c2202-98944d8f9-vxvvs"
Oct 30 13:20:39.187075 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:39.184510    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m2ch\" (UniqueName: \"kubernetes.io/projected/3330a40e-0641-4286-a850-f37c0b25110d-kube-api-access-8m2ch\") pod \"maas-api-e2e-api-a-3c2202-98944d8f9-vxvvs\" (UID: \"3330a40e-0641-4286-a850-f37c0b25110d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c2202-98944d8f9-vxvvs"
Oct 30 13:20:39.187075 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:39.184589    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3330a40e-0641-4286-a850-f37c0b25110d-openshift-service-ca\") pod \"maas-api-e2e-api-a-3c2202-98944d8f9-vxvvs\" (UID: \"3330a40e-0641-4286-a850-f37c0b25110d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c2202-98944d8f9-vxvvs"
Oct 30 13:20:39.187075 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:39.184661    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3330a40e-0641-4286-a850-f37c0b25110d-metrics-certs\") pod \"maas-api-e2e-api-a-3c2202-98944d8f9-vxvvs\" (UID: \"3330a40e-0641-4286-a850-f37c0b25110d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c2202-98944d8f9-vxvvs"
Oct 30 13:20:39.187075 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:39.184760    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3330a40e-0641-4286-a850-f37c0b25110d-maas-api-tls\") pod \"maas-api-e2e-api-a-3c2202-98944d8f9-vxvvs\" (UID: \"3330a40e-0641-4286-a850-f37c0b25110d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c2202-98944d8f9-vxvvs"
Oct 30 13:20:39.190260 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:39.189489    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-3c2202-serving-cert\""
Oct 30 13:20:39.286422 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:39.285846    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3330a40e-0641-4286-a850-f37c0b25110d-maas-api-tls\") pod \"maas-api-e2e-api-a-3c2202-98944d8f9-vxvvs\" (UID: \"3330a40e-0641-4286-a850-f37c0b25110d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c2202-98944d8f9-vxvvs"
Oct 30 13:20:39.286422 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:20:39.286284    5093 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-3c2202-serving-cert: secret "maas-api-e2e-api-a-3c2202-serving-cert" not found
Oct 30 13:20:39.286422 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:20:39.286376    5093 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3330a40e-0641-4286-a850-f37c0b25110d-maas-api-tls podName:3330a40e-0641-4286-a850-f37c0b25110d nodeName:}" failed. No retries permitted until 2026-10-30 13:20:39.786351078 +0000 UTC m=+1058.933618300 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3330a40e-0641-4286-a850-f37c0b25110d-maas-api-tls") pod "maas-api-e2e-api-a-3c2202-98944d8f9-vxvvs" (UID: "3330a40e-0641-4286-a850-f37c0b25110d") : secret "maas-api-e2e-api-a-3c2202-serving-cert" not found
Oct 30 13:20:39.291797 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:39.291118    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m2ch\" (UniqueName: \"kubernetes.io/projected/3330a40e-0641-4286-a850-f37c0b25110d-kube-api-access-8m2ch\") pod \"maas-api-e2e-api-a-3c2202-98944d8f9-vxvvs\" (UID: \"3330a40e-0641-4286-a850-f37c0b25110d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c2202-98944d8f9-vxvvs"
Oct 30 13:20:39.291797 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:39.291224    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3330a40e-0641-4286-a850-f37c0b25110d-openshift-service-ca\") pod \"maas-api-e2e-api-a-3c2202-98944d8f9-vxvvs\" (UID: \"3330a40e-0641-4286-a850-f37c0b25110d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c2202-98944d8f9-vxvvs"
Oct 30 13:20:39.291797 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:39.291315    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3330a40e-0641-4286-a850-f37c0b25110d-metrics-certs\") pod \"maas-api-e2e-api-a-3c2202-98944d8f9-vxvvs\" (UID: \"3330a40e-0641-4286-a850-f37c0b25110d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c2202-98944d8f9-vxvvs"
Oct 30 13:20:39.292548 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:39.292521    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3330a40e-0641-4286-a850-f37c0b25110d-openshift-service-ca\") pod \"maas-api-e2e-api-a-3c2202-98944d8f9-vxvvs\" (UID: \"3330a40e-0641-4286-a850-f37c0b25110d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c2202-98944d8f9-vxvvs"
Oct 30 13:20:39.294961 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:39.294931    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3330a40e-0641-4286-a850-f37c0b25110d-metrics-certs\") pod \"maas-api-e2e-api-a-3c2202-98944d8f9-vxvvs\" (UID: \"3330a40e-0641-4286-a850-f37c0b25110d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c2202-98944d8f9-vxvvs"
Oct 30 13:20:39.315859 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:39.315819    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m2ch\" (UniqueName: \"kubernetes.io/projected/3330a40e-0641-4286-a850-f37c0b25110d-kube-api-access-8m2ch\") pod \"maas-api-e2e-api-a-3c2202-98944d8f9-vxvvs\" (UID: \"3330a40e-0641-4286-a850-f37c0b25110d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c2202-98944d8f9-vxvvs"
Oct 30 13:20:39.798751 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:39.798712    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3330a40e-0641-4286-a850-f37c0b25110d-maas-api-tls\") pod \"maas-api-e2e-api-a-3c2202-98944d8f9-vxvvs\" (UID: \"3330a40e-0641-4286-a850-f37c0b25110d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c2202-98944d8f9-vxvvs"
Oct 30 13:20:39.801264 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:39.801239    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3330a40e-0641-4286-a850-f37c0b25110d-maas-api-tls\") pod \"maas-api-e2e-api-a-3c2202-98944d8f9-vxvvs\" (UID: \"3330a40e-0641-4286-a850-f37c0b25110d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c2202-98944d8f9-vxvvs"
Oct 30 13:20:39.805309 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:39.805287    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c2202-98944d8f9-vxvvs"
Oct 30 13:20:40.119166 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:40.119110    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3c2202-98944d8f9-vxvvs"]
Oct 30 13:20:40.120291 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:20:40.120265    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3330a40e_0641_4286_a850_f37c0b25110d.slice/crio-5762e89b6d82394a14324fd74f997f91f5a3fc625ece795e34c295bccc4198c7 WatchSource:0}: Error finding container 5762e89b6d82394a14324fd74f997f91f5a3fc625ece795e34c295bccc4198c7: Status 404 returned error can't find the container with id 5762e89b6d82394a14324fd74f997f91f5a3fc625ece795e34c295bccc4198c7
Oct 30 13:20:40.502190 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:40.502156    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c2202-98944d8f9-vxvvs" event={"ID":"3330a40e-0641-4286-a850-f37c0b25110d","Type":"ContainerStarted","Data":"5762e89b6d82394a14324fd74f997f91f5a3fc625ece795e34c295bccc4198c7"}
Oct 30 13:20:41.512422 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:41.512382    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c2202-98944d8f9-vxvvs" event={"ID":"3330a40e-0641-4286-a850-f37c0b25110d","Type":"ContainerStarted","Data":"076cca7837bb30d2da69f3f35ee8eeb3ee8ca253ba90ec04f6f82999128e229b"}
Oct 30 13:20:41.512928 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:41.512682    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c2202-98944d8f9-vxvvs"
Oct 30 13:20:41.567569 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:41.567506    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c2202-98944d8f9-vxvvs" podStartSLOduration=2.135609 podStartE2EDuration="2.567489809s" totalImagesPullingTime="431.880809ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:41.563708999 +0000 UTC m=+1060.710976239" watchObservedRunningTime="2026-10-30 13:20:41.567489809 +0000 UTC m=+1060.714757050"
Oct 30 13:20:47.526985 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:47.526950    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c2202-98944d8f9-vxvvs"
Oct 30 13:20:51.167723 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:20:51.167683    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:20:59.081460 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:59.081421    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4"]
Oct 30 13:20:59.086764 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:59.086736    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4"
Oct 30 13:20:59.105649 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:59.105619    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-7c0e28-openshift-default-dockercfg-8wqrc\""
Oct 30 13:20:59.123886 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:59.123855    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4"]
Oct 30 13:20:59.259517 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:59.259377    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e9f2890-8091-4bd7-8471-9c7d97081bd3-workload-certs\") pod \"e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4\" (UID: \"6e9f2890-8091-4bd7-8471-9c7d97081bd3\") " pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4"
Oct 30 13:20:59.260520 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:59.260490    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e9f2890-8091-4bd7-8471-9c7d97081bd3-istiod-ca-cert\") pod \"e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4\" (UID: \"6e9f2890-8091-4bd7-8471-9c7d97081bd3\") " pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4"
Oct 30 13:20:59.260786 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:59.260751    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e9f2890-8091-4bd7-8471-9c7d97081bd3-istio-token\") pod \"e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4\" (UID: \"6e9f2890-8091-4bd7-8471-9c7d97081bd3\") " pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4"
Oct 30 13:20:59.261000 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:59.260981    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnlqf\" (UniqueName: \"kubernetes.io/projected/6e9f2890-8091-4bd7-8471-9c7d97081bd3-kube-api-access-nnlqf\") pod \"e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4\" (UID: \"6e9f2890-8091-4bd7-8471-9c7d97081bd3\") " pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4"
Oct 30 13:20:59.261237 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:59.261221    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e9f2890-8091-4bd7-8471-9c7d97081bd3-credential-socket\") pod \"e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4\" (UID: \"6e9f2890-8091-4bd7-8471-9c7d97081bd3\") " pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4"
Oct 30 13:20:59.261392 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:59.261379    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e9f2890-8091-4bd7-8471-9c7d97081bd3-istio-podinfo\") pod \"e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4\" (UID: \"6e9f2890-8091-4bd7-8471-9c7d97081bd3\") " pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4"
Oct 30 13:20:59.262156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:59.262122    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e9f2890-8091-4bd7-8471-9c7d97081bd3-workload-socket\") pod \"e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4\" (UID: \"6e9f2890-8091-4bd7-8471-9c7d97081bd3\") " pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4"
Oct 30 13:20:59.262372 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:59.262356    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e9f2890-8091-4bd7-8471-9c7d97081bd3-istio-data\") pod \"e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4\" (UID: \"6e9f2890-8091-4bd7-8471-9c7d97081bd3\") " pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4"
Oct 30 13:20:59.262563 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:59.262549    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e9f2890-8091-4bd7-8471-9c7d97081bd3-istio-envoy\") pod \"e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4\" (UID: \"6e9f2890-8091-4bd7-8471-9c7d97081bd3\") " pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4"
Oct 30 13:20:59.365530 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:59.365422    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnlqf\" (UniqueName: \"kubernetes.io/projected/6e9f2890-8091-4bd7-8471-9c7d97081bd3-kube-api-access-nnlqf\") pod \"e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4\" (UID: \"6e9f2890-8091-4bd7-8471-9c7d97081bd3\") " pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4"
Oct 30 13:20:59.365530 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:59.365487    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e9f2890-8091-4bd7-8471-9c7d97081bd3-credential-socket\") pod \"e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4\" (UID: \"6e9f2890-8091-4bd7-8471-9c7d97081bd3\") " pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4"
Oct 30 13:20:59.365530 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:59.365524    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e9f2890-8091-4bd7-8471-9c7d97081bd3-istio-podinfo\") pod \"e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4\" (UID: \"6e9f2890-8091-4bd7-8471-9c7d97081bd3\") " pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4"
Oct 30 13:20:59.365806 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:59.365598    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e9f2890-8091-4bd7-8471-9c7d97081bd3-workload-socket\") pod \"e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4\" (UID: \"6e9f2890-8091-4bd7-8471-9c7d97081bd3\") " pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4"
Oct 30 13:20:59.365806 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:59.365665    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e9f2890-8091-4bd7-8471-9c7d97081bd3-istio-data\") pod \"e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4\" (UID: \"6e9f2890-8091-4bd7-8471-9c7d97081bd3\") " pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4"
Oct 30 13:20:59.365806 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:59.365721    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e9f2890-8091-4bd7-8471-9c7d97081bd3-istio-envoy\") pod \"e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4\" (UID: \"6e9f2890-8091-4bd7-8471-9c7d97081bd3\") " pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4"
Oct 30 13:20:59.365978 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:59.365958    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e9f2890-8091-4bd7-8471-9c7d97081bd3-workload-certs\") pod \"e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4\" (UID: \"6e9f2890-8091-4bd7-8471-9c7d97081bd3\") " pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4"
Oct 30 13:20:59.367100 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:59.367074    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e9f2890-8091-4bd7-8471-9c7d97081bd3-istiod-ca-cert\") pod \"e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4\" (UID: \"6e9f2890-8091-4bd7-8471-9c7d97081bd3\") " pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4"
Oct 30 13:20:59.367334 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:59.367313    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e9f2890-8091-4bd7-8471-9c7d97081bd3-istio-token\") pod \"e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4\" (UID: \"6e9f2890-8091-4bd7-8471-9c7d97081bd3\") " pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4"
Oct 30 13:20:59.368220 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:59.368198    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e9f2890-8091-4bd7-8471-9c7d97081bd3-workload-certs\") pod \"e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4\" (UID: \"6e9f2890-8091-4bd7-8471-9c7d97081bd3\") " pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4"
Oct 30 13:20:59.368659 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:59.368639    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e9f2890-8091-4bd7-8471-9c7d97081bd3-workload-socket\") pod \"e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4\" (UID: \"6e9f2890-8091-4bd7-8471-9c7d97081bd3\") " pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4"
Oct 30 13:20:59.369012 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:59.368992    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e9f2890-8091-4bd7-8471-9c7d97081bd3-istio-data\") pod \"e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4\" (UID: \"6e9f2890-8091-4bd7-8471-9c7d97081bd3\") " pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4"
Oct 30 13:20:59.370085 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:59.370053    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e9f2890-8091-4bd7-8471-9c7d97081bd3-istiod-ca-cert\") pod \"e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4\" (UID: \"6e9f2890-8091-4bd7-8471-9c7d97081bd3\") " pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4"
Oct 30 13:20:59.370804 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:59.370782    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e9f2890-8091-4bd7-8471-9c7d97081bd3-credential-socket\") pod \"e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4\" (UID: \"6e9f2890-8091-4bd7-8471-9c7d97081bd3\") " pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4"
Oct 30 13:20:59.373298 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:59.372725    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e9f2890-8091-4bd7-8471-9c7d97081bd3-istio-podinfo\") pod \"e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4\" (UID: \"6e9f2890-8091-4bd7-8471-9c7d97081bd3\") " pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4"
Oct 30 13:20:59.373298 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:59.372976    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e9f2890-8091-4bd7-8471-9c7d97081bd3-istio-envoy\") pod \"e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4\" (UID: \"6e9f2890-8091-4bd7-8471-9c7d97081bd3\") " pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4"
Oct 30 13:20:59.390024 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:59.389977    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e9f2890-8091-4bd7-8471-9c7d97081bd3-istio-token\") pod \"e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4\" (UID: \"6e9f2890-8091-4bd7-8471-9c7d97081bd3\") " pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4"
Oct 30 13:20:59.393604 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:59.393571    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnlqf\" (UniqueName: \"kubernetes.io/projected/6e9f2890-8091-4bd7-8471-9c7d97081bd3-kube-api-access-nnlqf\") pod \"e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4\" (UID: \"6e9f2890-8091-4bd7-8471-9c7d97081bd3\") " pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4"
Oct 30 13:20:59.407479 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:59.407435    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4"
Oct 30 13:20:59.816521 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:59.816465    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4"]
Oct 30 13:20:59.849972 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:59.849897    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:20:59.864165 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:59.855289    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:20:59.864165 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:20:59.855364    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:21:00.698993 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:00.698942    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4" event={"ID":"6e9f2890-8091-4bd7-8471-9c7d97081bd3","Type":"ContainerStarted","Data":"47fc90b4cc96de627928b8490fac0061319075dafcb5d66cd2d7eb5ee2177df1"}
Oct 30 13:21:00.699569 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:00.699013    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4" event={"ID":"6e9f2890-8091-4bd7-8471-9c7d97081bd3","Type":"ContainerStarted","Data":"2a45db66db571f860408fbb15cb9b24ff16cf112651f31f0826cd870ff00bdf9"}
Oct 30 13:21:00.777155 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:00.776703    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4" podStartSLOduration=1.77668076 podStartE2EDuration="1.77668076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:00.760790292 +0000 UTC m=+1079.908057533" watchObservedRunningTime="2026-10-30 13:21:00.77668076 +0000 UTC m=+1079.923948002"
Oct 30 13:21:01.407799 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:01.407761    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4"
Oct 30 13:21:01.418832 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:01.418798    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4"
Oct 30 13:21:01.709061 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:01.709026    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4"
Oct 30 13:21:01.710445 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:01.710422    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4"
Oct 30 13:21:09.716775 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:09.716738    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2ed395-56fd58558-jwsg9"]
Oct 30 13:21:09.717267 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:09.716999    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ed395-56fd58558-jwsg9" podUID="99d6ebaa-4374-4502-90db-59d12b6836c0" containerName="maas-api" containerID="cri-o://34c3d488e0b5adf17b057ab5282614eb479fdd80b7f42be827f433c0dfa3b31c" gracePeriod=30
Oct 30 13:21:10.127487 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:10.127453    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ed395-56fd58558-jwsg9"
Oct 30 13:21:10.241121 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:10.241081    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99d6ebaa-4374-4502-90db-59d12b6836c0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/99d6ebaa-4374-4502-90db-59d12b6836c0-metrics-certs\") pod \"99d6ebaa-4374-4502-90db-59d12b6836c0\" (UID: \"99d6ebaa-4374-4502-90db-59d12b6836c0\") "
Oct 30 13:21:10.241349 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:10.241202    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99d6ebaa-4374-4502-90db-59d12b6836c0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99d6ebaa-4374-4502-90db-59d12b6836c0-openshift-service-ca\") pod \"99d6ebaa-4374-4502-90db-59d12b6836c0\" (UID: \"99d6ebaa-4374-4502-90db-59d12b6836c0\") "
Oct 30 13:21:10.241349 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:10.241301    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99d6ebaa-4374-4502-90db-59d12b6836c0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99d6ebaa-4374-4502-90db-59d12b6836c0-maas-api-tls\") pod \"99d6ebaa-4374-4502-90db-59d12b6836c0\" (UID: \"99d6ebaa-4374-4502-90db-59d12b6836c0\") "
Oct 30 13:21:10.241453 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:10.241367    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99d6ebaa-4374-4502-90db-59d12b6836c0-kube-api-access-9t9b6\" (UniqueName: \"kubernetes.io/projected/99d6ebaa-4374-4502-90db-59d12b6836c0-kube-api-access-9t9b6\") pod \"99d6ebaa-4374-4502-90db-59d12b6836c0\" (UID: \"99d6ebaa-4374-4502-90db-59d12b6836c0\") "
Oct 30 13:21:10.244718 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:10.244684    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99d6ebaa-4374-4502-90db-59d12b6836c0-kube-api-access-9t9b6" pod "99d6ebaa-4374-4502-90db-59d12b6836c0" (UID: "99d6ebaa-4374-4502-90db-59d12b6836c0"). InnerVolumeSpecName "kube-api-access-9t9b6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:21:10.245968 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:10.245937    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99d6ebaa-4374-4502-90db-59d12b6836c0-openshift-service-ca" pod "99d6ebaa-4374-4502-90db-59d12b6836c0" (UID: "99d6ebaa-4374-4502-90db-59d12b6836c0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:21:10.249356 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:10.249329    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99d6ebaa-4374-4502-90db-59d12b6836c0-metrics-certs" pod "99d6ebaa-4374-4502-90db-59d12b6836c0" (UID: "99d6ebaa-4374-4502-90db-59d12b6836c0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:21:10.252806 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:10.252778    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99d6ebaa-4374-4502-90db-59d12b6836c0-maas-api-tls" pod "99d6ebaa-4374-4502-90db-59d12b6836c0" (UID: "99d6ebaa-4374-4502-90db-59d12b6836c0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:21:10.344236 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:10.344070    5093 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99d6ebaa-4374-4502-90db-59d12b6836c0-maas-api-tls\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:21:10.344236 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:10.344105    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t9b6\" (UniqueName: \"kubernetes.io/projected/99d6ebaa-4374-4502-90db-59d12b6836c0-kube-api-access-9t9b6\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:21:10.344236 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:10.344116    5093 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99d6ebaa-4374-4502-90db-59d12b6836c0-metrics-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:21:10.344236 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:10.344125    5093 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99d6ebaa-4374-4502-90db-59d12b6836c0-openshift-service-ca\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:21:10.806937 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:10.806894    5093 generic.go:356] "Generic (PLEG): container finished" podID="99d6ebaa-4374-4502-90db-59d12b6836c0" containerID="34c3d488e0b5adf17b057ab5282614eb479fdd80b7f42be827f433c0dfa3b31c" exitCode=0
Oct 30 13:21:10.807453 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:10.807010    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ed395-56fd58558-jwsg9" event={"ID":"99d6ebaa-4374-4502-90db-59d12b6836c0","Type":"ContainerDied","Data":"34c3d488e0b5adf17b057ab5282614eb479fdd80b7f42be827f433c0dfa3b31c"}
Oct 30 13:21:10.807453 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:10.807115    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ed395-56fd58558-jwsg9" event={"ID":"99d6ebaa-4374-4502-90db-59d12b6836c0","Type":"ContainerDied","Data":"292a6da663df33b2ae2b771fbe8106c8f17c63a2e95b1b60d39f4d4073f9a2c8"}
Oct 30 13:21:10.807453 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:10.807186    5093 scope.go:122] "RemoveContainer" containerID="34c3d488e0b5adf17b057ab5282614eb479fdd80b7f42be827f433c0dfa3b31c"
Oct 30 13:21:10.807453 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:10.807329    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ed395-56fd58558-jwsg9"
Oct 30 13:21:10.820224 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:10.820114    5093 scope.go:122] "RemoveContainer" containerID="34c3d488e0b5adf17b057ab5282614eb479fdd80b7f42be827f433c0dfa3b31c"
Oct 30 13:21:10.820493 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:21:10.820469    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34c3d488e0b5adf17b057ab5282614eb479fdd80b7f42be827f433c0dfa3b31c\": container with ID starting with 34c3d488e0b5adf17b057ab5282614eb479fdd80b7f42be827f433c0dfa3b31c not found: ID does not exist" containerID="34c3d488e0b5adf17b057ab5282614eb479fdd80b7f42be827f433c0dfa3b31c"
Oct 30 13:21:10.820573 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:10.820511    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34c3d488e0b5adf17b057ab5282614eb479fdd80b7f42be827f433c0dfa3b31c"} err="failed to get container status \"34c3d488e0b5adf17b057ab5282614eb479fdd80b7f42be827f433c0dfa3b31c\": rpc error: code = NotFound desc = could not find container \"34c3d488e0b5adf17b057ab5282614eb479fdd80b7f42be827f433c0dfa3b31c\": container with ID starting with 34c3d488e0b5adf17b057ab5282614eb479fdd80b7f42be827f433c0dfa3b31c not found: ID does not exist"
Oct 30 13:21:10.865424 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:10.865388    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2ed395-56fd58558-jwsg9"]
Oct 30 13:21:10.877866 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:10.877836    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2ed395-56fd58558-jwsg9"]
Oct 30 13:21:11.003987 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:11.003956    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99d6ebaa-4374-4502-90db-59d12b6836c0" path="/var/lib/kubelet/pods/99d6ebaa-4374-4502-90db-59d12b6836c0/volumes"
Oct 30 13:21:20.639894 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:20.639852    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c7vfw"]
Oct 30 13:21:20.640931 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:20.640905    5093 cpu_manager.go:414] "containerMap: removing container" podUID="99d6ebaa-4374-4502-90db-59d12b6836c0" containerName="maas-api"
Oct 30 13:21:20.640993 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:20.640934    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99d6ebaa-4374-4502-90db-59d12b6836c0" containerName="maas-api"
Oct 30 13:21:20.641165 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:20.641151    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="99d6ebaa-4374-4502-90db-59d12b6836c0" containerName="maas-api"
Oct 30 13:21:20.645794 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:20.645771    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c7vfw"
Oct 30 13:21:20.666173 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:20.666143    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c7vfw"]
Oct 30 13:21:20.673201 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:20.673008    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 13:21:20.673201 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:20.673086    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-fqpnj\""
Oct 30 13:21:20.673201 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:20.673098    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 13:21:20.673201 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:20.673189    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 13:21:20.743553 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:20.743499    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/731d4cfd-f9fb-4f4c-bca1-4b632cd67156-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c7vfw\" (UID: \"731d4cfd-f9fb-4f4c-bca1-4b632cd67156\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c7vfw"
Oct 30 13:21:20.743709 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:20.743641    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s44xv\" (UniqueName: \"kubernetes.io/projected/731d4cfd-f9fb-4f4c-bca1-4b632cd67156-kube-api-access-s44xv\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c7vfw\" (UID: \"731d4cfd-f9fb-4f4c-bca1-4b632cd67156\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c7vfw"
Oct 30 13:21:20.743822 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:20.743804    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/731d4cfd-f9fb-4f4c-bca1-4b632cd67156-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c7vfw\" (UID: \"731d4cfd-f9fb-4f4c-bca1-4b632cd67156\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c7vfw"
Oct 30 13:21:20.743874 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:20.743858    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/731d4cfd-f9fb-4f4c-bca1-4b632cd67156-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c7vfw\" (UID: \"731d4cfd-f9fb-4f4c-bca1-4b632cd67156\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c7vfw"
Oct 30 13:21:20.743915 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:20.743897    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/731d4cfd-f9fb-4f4c-bca1-4b632cd67156-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c7vfw\" (UID: \"731d4cfd-f9fb-4f4c-bca1-4b632cd67156\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c7vfw"
Oct 30 13:21:20.744039 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:20.744013    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/731d4cfd-f9fb-4f4c-bca1-4b632cd67156-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c7vfw\" (UID: \"731d4cfd-f9fb-4f4c-bca1-4b632cd67156\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c7vfw"
Oct 30 13:21:20.744152 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:20.744103    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/731d4cfd-f9fb-4f4c-bca1-4b632cd67156-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c7vfw\" (UID: \"731d4cfd-f9fb-4f4c-bca1-4b632cd67156\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c7vfw"
Oct 30 13:21:20.845333 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:20.845291    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s44xv\" (UniqueName: \"kubernetes.io/projected/731d4cfd-f9fb-4f4c-bca1-4b632cd67156-kube-api-access-s44xv\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c7vfw\" (UID: \"731d4cfd-f9fb-4f4c-bca1-4b632cd67156\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c7vfw"
Oct 30 13:21:20.845532 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:20.845434    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/731d4cfd-f9fb-4f4c-bca1-4b632cd67156-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c7vfw\" (UID: \"731d4cfd-f9fb-4f4c-bca1-4b632cd67156\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c7vfw"
Oct 30 13:21:20.845532 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:20.845469    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/731d4cfd-f9fb-4f4c-bca1-4b632cd67156-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c7vfw\" (UID: \"731d4cfd-f9fb-4f4c-bca1-4b632cd67156\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c7vfw"
Oct 30 13:21:20.845532 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:20.845506    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/731d4cfd-f9fb-4f4c-bca1-4b632cd67156-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c7vfw\" (UID: \"731d4cfd-f9fb-4f4c-bca1-4b632cd67156\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c7vfw"
Oct 30 13:21:20.845703 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:20.845593    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/731d4cfd-f9fb-4f4c-bca1-4b632cd67156-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c7vfw\" (UID: \"731d4cfd-f9fb-4f4c-bca1-4b632cd67156\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c7vfw"
Oct 30 13:21:20.845703 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:20.845665    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/731d4cfd-f9fb-4f4c-bca1-4b632cd67156-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c7vfw\" (UID: \"731d4cfd-f9fb-4f4c-bca1-4b632cd67156\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c7vfw"
Oct 30 13:21:20.845806 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:20.845775    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/731d4cfd-f9fb-4f4c-bca1-4b632cd67156-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c7vfw\" (UID: \"731d4cfd-f9fb-4f4c-bca1-4b632cd67156\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c7vfw"
Oct 30 13:21:20.846602 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:20.846573    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/731d4cfd-f9fb-4f4c-bca1-4b632cd67156-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c7vfw\" (UID: \"731d4cfd-f9fb-4f4c-bca1-4b632cd67156\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c7vfw"
Oct 30 13:21:20.848418 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:20.848390    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/731d4cfd-f9fb-4f4c-bca1-4b632cd67156-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c7vfw\" (UID: \"731d4cfd-f9fb-4f4c-bca1-4b632cd67156\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c7vfw"
Oct 30 13:21:20.848418 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:20.848411    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/731d4cfd-f9fb-4f4c-bca1-4b632cd67156-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c7vfw\" (UID: \"731d4cfd-f9fb-4f4c-bca1-4b632cd67156\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c7vfw"
Oct 30 13:21:20.848620 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:20.848597    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/731d4cfd-f9fb-4f4c-bca1-4b632cd67156-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c7vfw\" (UID: \"731d4cfd-f9fb-4f4c-bca1-4b632cd67156\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c7vfw"
Oct 30 13:21:20.848941 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:20.848920    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/731d4cfd-f9fb-4f4c-bca1-4b632cd67156-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c7vfw\" (UID: \"731d4cfd-f9fb-4f4c-bca1-4b632cd67156\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c7vfw"
Oct 30 13:21:20.866355 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:20.866311    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s44xv\" (UniqueName: \"kubernetes.io/projected/731d4cfd-f9fb-4f4c-bca1-4b632cd67156-kube-api-access-s44xv\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c7vfw\" (UID: \"731d4cfd-f9fb-4f4c-bca1-4b632cd67156\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c7vfw"
Oct 30 13:21:20.866520 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:20.866401    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/731d4cfd-f9fb-4f4c-bca1-4b632cd67156-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c7vfw\" (UID: \"731d4cfd-f9fb-4f4c-bca1-4b632cd67156\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c7vfw"
Oct 30 13:21:20.965149 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:20.965104    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c7vfw"
Oct 30 13:21:21.170979 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:21:21.170936    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:21:21.288183 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:21.288083    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c7vfw"]
Oct 30 13:21:21.297752 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:21:21.297723    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod731d4cfd_f9fb_4f4c_bca1_4b632cd67156.slice/crio-39c1615f150e291b8e7c9359140780fe3483dcf53ea412dc8bb07fad105d1aa1 WatchSource:0}: Error finding container 39c1615f150e291b8e7c9359140780fe3483dcf53ea412dc8bb07fad105d1aa1: Status 404 returned error can't find the container with id 39c1615f150e291b8e7c9359140780fe3483dcf53ea412dc8bb07fad105d1aa1
Oct 30 13:21:21.990447 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:21.990405    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c7vfw" event={"ID":"731d4cfd-f9fb-4f4c-bca1-4b632cd67156","Type":"ContainerStarted","Data":"39c1615f150e291b8e7c9359140780fe3483dcf53ea412dc8bb07fad105d1aa1"}
Oct 30 13:21:25.638159 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:25.630635    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:21:25.638159 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:25.630723    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:21:26.110920 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:26.110884    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c7vfw" event={"ID":"731d4cfd-f9fb-4f4c-bca1-4b632cd67156","Type":"ContainerStarted","Data":"c7162acd6aac34553810cea785bcdf3e473101bf0e48bfc83470a5eef5ef5342"}
Oct 30 13:21:26.111170 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:26.110947    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c7vfw"
Oct 30 13:21:26.113695 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:26.113664    5093 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-c7vfw container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:21:26.113841 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:26.113719    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c7vfw" podUID="731d4cfd-f9fb-4f4c-bca1-4b632cd67156" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:21:26.201822 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:26.201770    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c7vfw" podStartSLOduration=1.8726436359999998 podStartE2EDuration="6.201756536s" totalImagesPullingTime="4.3291129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:26.199401083 +0000 UTC m=+1105.346668324" watchObservedRunningTime="2026-10-30 13:21:26.201756536 +0000 UTC m=+1105.349023776"
Oct 30 13:21:27.123903 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:27.123874    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c7vfw"
Oct 30 13:21:33.219202 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:33.219124    5093 generic.go:356] "Generic (PLEG): container finished" podID="c0b3e685-671d-4227-bd45-f3a614caf44d" containerID="941f9890e9b596c68bee2804faf8cf39e419581aac827f00c4f546dd06362a9b" exitCode=0
Oct 30 13:21:33.219824 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:33.219204    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb" event={"ID":"c0b3e685-671d-4227-bd45-f3a614caf44d","Type":"ContainerDied","Data":"941f9890e9b596c68bee2804faf8cf39e419581aac827f00c4f546dd06362a9b"}
Oct 30 13:21:33.219824 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:33.219668    5093 scope.go:122] "RemoveContainer" containerID="941f9890e9b596c68bee2804faf8cf39e419581aac827f00c4f546dd06362a9b"
Oct 30 13:21:33.221257 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:33.221231    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:21:33.221339 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:33.221325    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:21:33.221381 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:33.221360    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:21:34.237097 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:34.237057    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb" event={"ID":"c0b3e685-671d-4227-bd45-f3a614caf44d","Type":"ContainerStarted","Data":"48061d2a25a431f7103514b13636123c2f29daa552e605f4064ee7ba4a94660b"}
Oct 30 13:21:34.723185 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:34.720799    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb"
Oct 30 13:21:34.723185 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:34.722365    5093 patch_prober.go:29] interesting pod/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 30 13:21:34.723185 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:34.722446    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb" podUID="c0b3e685-671d-4227-bd45-f3a614caf44d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 30 13:21:35.211998 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:35.211966    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb"
Oct 30 13:21:35.721101 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:35.721065    5093 patch_prober.go:29] interesting pod/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 30 13:21:35.721539 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:35.721118    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb" podUID="c0b3e685-671d-4227-bd45-f3a614caf44d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 30 13:21:36.829200 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:36.828913    5093 patch_prober.go:29] interesting pod/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 30 13:21:36.829200 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:36.828976    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb" podUID="c0b3e685-671d-4227-bd45-f3a614caf44d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 30 13:21:37.774154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:37.766276    5093 patch_prober.go:29] interesting pod/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 30 13:21:37.774154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:37.766360    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb" podUID="c0b3e685-671d-4227-bd45-f3a614caf44d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 30 13:21:38.727399 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:38.727361    5093 patch_prober.go:29] interesting pod/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 30 13:21:38.728015 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:38.727981    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb" podUID="c0b3e685-671d-4227-bd45-f3a614caf44d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 30 13:21:39.751686 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:39.723265    5093 patch_prober.go:29] interesting pod/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 30 13:21:39.751686 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:39.739066    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb" podUID="c0b3e685-671d-4227-bd45-f3a614caf44d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 30 13:21:40.420620 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:40.420564    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv" podUID="d6a44981-104f-4d3e-9937-d184a6ddc482" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 13:21:40.727033 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:40.726988    5093 patch_prober.go:29] interesting pod/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 30 13:21:40.727367 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:40.727327    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb" podUID="c0b3e685-671d-4227-bd45-f3a614caf44d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 30 13:21:40.975771 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:40.975714    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx" podUID="ba156eeb-9955-4f9a-857b-2c4d502a5fcc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:21:41.034344 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:41.034229    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt" podUID="4ae52c69-0c30-438f-9ac0-c87b6de16061" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:21:41.141487 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:41.141433    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg" podUID="ea079ac1-f502-4b5f-b16a-c91e2de98de8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:21:41.226780 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:41.226729    5093 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt" podUID="4ae52c69-0c30-438f-9ac0-c87b6de16061" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:21:41.456121 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:41.456073    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-6b7b93a3/e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5" podUID="81a50a32-4858-4cfc-94d0-754c8e89f6a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:21:41.749675 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:41.749584    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb"
Oct 30 13:21:41.752498 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:41.752473    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb"
Oct 30 13:21:42.098211 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:42.094189    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt" podUID="4ae52c69-0c30-438f-9ac0-c87b6de16061" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:21:42.215156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:42.214235    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt"
Oct 30 13:21:42.215347 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:42.142154    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg" podUID="ea079ac1-f502-4b5f-b16a-c91e2de98de8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Oct 30 13:21:42.234077 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:42.234043    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg"
Oct 30 13:21:42.528183 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:42.499363    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-6b7b93a3/e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5" podUID="81a50a32-4858-4cfc-94d0-754c8e89f6a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded"
Oct 30 13:21:42.528183 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:42.499934    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-6b7b93a3/e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5"
Oct 30 13:21:42.528183 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:42.524214    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25" podUID="7805f89f-89d4-4754-b606-c3070f4df215" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:21:42.528183 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:42.527674    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb" podUID="c4b4e00e-f58d-4a4e-b7db-19f724360db1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:21:42.593154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:42.591116    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg"
Oct 30 13:21:42.939154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:42.929901    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-6b7b93a3/e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5"
Oct 30 13:21:43.006439 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:43.004276    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx" podUID="ba156eeb-9955-4f9a-857b-2c4d502a5fcc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:21:43.232092 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:43.231987    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt" podUID="4ae52c69-0c30-438f-9ac0-c87b6de16061" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 30 13:21:43.315402 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:43.315370    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt"
Oct 30 13:21:47.803321 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:47.803277    5093 patch_prober.go:29] interesting pod/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:21:47.803965 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:47.803927    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4" podUID="6e9f2890-8091-4bd7-8471-9c7d97081bd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:21:51.169166 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:51.169095    5093 generic.go:356] "Generic (PLEG): container finished" podID="6e9f2890-8091-4bd7-8471-9c7d97081bd3" containerID="47fc90b4cc96de627928b8490fac0061319075dafcb5d66cd2d7eb5ee2177df1" exitCode=0
Oct 30 13:21:51.169660 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:51.169521    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4" event={"ID":"6e9f2890-8091-4bd7-8471-9c7d97081bd3","Type":"ContainerDied","Data":"47fc90b4cc96de627928b8490fac0061319075dafcb5d66cd2d7eb5ee2177df1"}
Oct 30 13:21:51.170084 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:51.170060    5093 scope.go:122] "RemoveContainer" containerID="47fc90b4cc96de627928b8490fac0061319075dafcb5d66cd2d7eb5ee2177df1"
Oct 30 13:21:51.172306 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:51.172267    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:21:51.172427 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:51.172367    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:21:51.172427 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:51.172414    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:21:51.179371 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:21:51.179325    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:21:52.208365 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:52.208331    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4" event={"ID":"6e9f2890-8091-4bd7-8471-9c7d97081bd3","Type":"ContainerStarted","Data":"adcda4545958767c492f93d5e4ba68df2e433b22b678cf156199a9ae57577d57"}
Oct 30 13:21:52.408745 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:52.408708    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4"
Oct 30 13:21:52.408928 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:52.408891    5093 patch_prober.go:29] interesting pod/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 13:21:52.408974 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:52.408952    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4" podUID="6e9f2890-8091-4bd7-8471-9c7d97081bd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 13:21:53.409059 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:53.408755    5093 patch_prober.go:29] interesting pod/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 13:21:53.409059 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:53.408812    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4" podUID="6e9f2890-8091-4bd7-8471-9c7d97081bd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 13:21:54.408306 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:54.408264    5093 patch_prober.go:29] interesting pod/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 13:21:54.408548 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:54.408323    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4" podUID="6e9f2890-8091-4bd7-8471-9c7d97081bd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 13:21:55.407966 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:55.407932    5093 patch_prober.go:29] interesting pod/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 13:21:55.408421 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:55.407998    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4" podUID="6e9f2890-8091-4bd7-8471-9c7d97081bd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 13:21:56.408043 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:56.408003    5093 patch_prober.go:29] interesting pod/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 13:21:56.408496 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:56.408078    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4" podUID="6e9f2890-8091-4bd7-8471-9c7d97081bd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 13:21:57.408305 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:57.408262    5093 patch_prober.go:29] interesting pod/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 13:21:57.408994 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:57.408957    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4" podUID="6e9f2890-8091-4bd7-8471-9c7d97081bd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 13:21:58.412912 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:58.412880    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4"
Oct 30 13:21:58.413416 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:58.413394    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4"
Oct 30 13:21:58.414259 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:21:58.414237    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4"
Oct 30 13:22:08.592965 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:08.592929    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-6b7b93a3/e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5"]
Oct 30 13:22:21.239422 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:22:21.239382    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:22:23.615664 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:23.615626    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-6b7b93a3/e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5" podUID="81a50a32-4858-4cfc-94d0-754c8e89f6a4" containerName="main" containerID="cri-o://929ac8585ac4e7d4661e9d2ea4c9eff0cd71ec823f2fbf290a82393d1c7dfcd5" gracePeriod=45
Oct 30 13:22:23.996649 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:23.996614    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6b7b93a3/e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5"
Oct 30 13:22:24.078710 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:24.078678    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81a50a32-4858-4cfc-94d0-754c8e89f6a4-kube-api-access-bnm98\" (UniqueName: \"kubernetes.io/projected/81a50a32-4858-4cfc-94d0-754c8e89f6a4-kube-api-access-bnm98\") pod \"81a50a32-4858-4cfc-94d0-754c8e89f6a4\" (UID: \"81a50a32-4858-4cfc-94d0-754c8e89f6a4\") "
Oct 30 13:22:24.078946 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:24.078926    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81a50a32-4858-4cfc-94d0-754c8e89f6a4-home\" (UniqueName: \"kubernetes.io/empty-dir/81a50a32-4858-4cfc-94d0-754c8e89f6a4-home\") pod \"81a50a32-4858-4cfc-94d0-754c8e89f6a4\" (UID: \"81a50a32-4858-4cfc-94d0-754c8e89f6a4\") "
Oct 30 13:22:24.079082 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:24.079068    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81a50a32-4858-4cfc-94d0-754c8e89f6a4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81a50a32-4858-4cfc-94d0-754c8e89f6a4-model-cache\") pod \"81a50a32-4858-4cfc-94d0-754c8e89f6a4\" (UID: \"81a50a32-4858-4cfc-94d0-754c8e89f6a4\") "
Oct 30 13:22:24.079208 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:24.079163    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81a50a32-4858-4cfc-94d0-754c8e89f6a4-tls-certs\" (UniqueName: \"kubernetes.io/secret/81a50a32-4858-4cfc-94d0-754c8e89f6a4-tls-certs\") pod \"81a50a32-4858-4cfc-94d0-754c8e89f6a4\" (UID: \"81a50a32-4858-4cfc-94d0-754c8e89f6a4\") "
Oct 30 13:22:24.079278 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:24.079231    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a50a32-4858-4cfc-94d0-754c8e89f6a4-home" pod "81a50a32-4858-4cfc-94d0-754c8e89f6a4" (UID: "81a50a32-4858-4cfc-94d0-754c8e89f6a4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:22:24.079381 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:24.079356    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a50a32-4858-4cfc-94d0-754c8e89f6a4-model-cache" pod "81a50a32-4858-4cfc-94d0-754c8e89f6a4" (UID: "81a50a32-4858-4cfc-94d0-754c8e89f6a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:22:24.081262 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:24.081233    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81a50a32-4858-4cfc-94d0-754c8e89f6a4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/81a50a32-4858-4cfc-94d0-754c8e89f6a4-dshm\") pod \"81a50a32-4858-4cfc-94d0-754c8e89f6a4\" (UID: \"81a50a32-4858-4cfc-94d0-754c8e89f6a4\") "
Oct 30 13:22:24.081453 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:24.081280    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81a50a32-4858-4cfc-94d0-754c8e89f6a4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81a50a32-4858-4cfc-94d0-754c8e89f6a4-tmp-dir\") pod \"81a50a32-4858-4cfc-94d0-754c8e89f6a4\" (UID: \"81a50a32-4858-4cfc-94d0-754c8e89f6a4\") "
Oct 30 13:22:24.082186 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:24.081834    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a50a32-4858-4cfc-94d0-754c8e89f6a4-tmp-dir" pod "81a50a32-4858-4cfc-94d0-754c8e89f6a4" (UID: "81a50a32-4858-4cfc-94d0-754c8e89f6a4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:22:24.082186 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:24.082000    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81a50a32-4858-4cfc-94d0-754c8e89f6a4-tls-certs" pod "81a50a32-4858-4cfc-94d0-754c8e89f6a4" (UID: "81a50a32-4858-4cfc-94d0-754c8e89f6a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:22:24.082366 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:24.082268    5093 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81a50a32-4858-4cfc-94d0-754c8e89f6a4-home\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:22:24.082366 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:24.082288    5093 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81a50a32-4858-4cfc-94d0-754c8e89f6a4-model-cache\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:22:24.082366 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:24.082304    5093 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81a50a32-4858-4cfc-94d0-754c8e89f6a4-tls-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:22:24.082366 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:24.082315    5093 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81a50a32-4858-4cfc-94d0-754c8e89f6a4-tmp-dir\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:22:24.084552 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:24.084520    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81a50a32-4858-4cfc-94d0-754c8e89f6a4-kube-api-access-bnm98" pod "81a50a32-4858-4cfc-94d0-754c8e89f6a4" (UID: "81a50a32-4858-4cfc-94d0-754c8e89f6a4"). InnerVolumeSpecName "kube-api-access-bnm98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:22:24.084756 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:24.084728    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a50a32-4858-4cfc-94d0-754c8e89f6a4-dshm" pod "81a50a32-4858-4cfc-94d0-754c8e89f6a4" (UID: "81a50a32-4858-4cfc-94d0-754c8e89f6a4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:22:24.184388 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:24.184337    5093 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81a50a32-4858-4cfc-94d0-754c8e89f6a4-dshm\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:22:24.184388 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:24.184387    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnm98\" (UniqueName: \"kubernetes.io/projected/81a50a32-4858-4cfc-94d0-754c8e89f6a4-kube-api-access-bnm98\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:22:24.578547 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:24.578506    5093 generic.go:356] "Generic (PLEG): container finished" podID="81a50a32-4858-4cfc-94d0-754c8e89f6a4" containerID="929ac8585ac4e7d4661e9d2ea4c9eff0cd71ec823f2fbf290a82393d1c7dfcd5" exitCode=0
Oct 30 13:22:24.578741 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:24.578558    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6b7b93a3/e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5"
Oct 30 13:22:24.578741 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:24.578590    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6b7b93a3/e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5" event={"ID":"81a50a32-4858-4cfc-94d0-754c8e89f6a4","Type":"ContainerDied","Data":"929ac8585ac4e7d4661e9d2ea4c9eff0cd71ec823f2fbf290a82393d1c7dfcd5"}
Oct 30 13:22:24.578741 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:24.578637    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6b7b93a3/e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5" event={"ID":"81a50a32-4858-4cfc-94d0-754c8e89f6a4","Type":"ContainerDied","Data":"178f355d8328bf9718adae7eafc44530652cad05d55d99eba60edbcfab522238"}
Oct 30 13:22:24.578741 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:24.578654    5093 scope.go:122] "RemoveContainer" containerID="929ac8585ac4e7d4661e9d2ea4c9eff0cd71ec823f2fbf290a82393d1c7dfcd5"
Oct 30 13:22:24.589901 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:24.589871    5093 scope.go:122] "RemoveContainer" containerID="929ac8585ac4e7d4661e9d2ea4c9eff0cd71ec823f2fbf290a82393d1c7dfcd5"
Oct 30 13:22:24.590240 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:22:24.590217    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"929ac8585ac4e7d4661e9d2ea4c9eff0cd71ec823f2fbf290a82393d1c7dfcd5\": container with ID starting with 929ac8585ac4e7d4661e9d2ea4c9eff0cd71ec823f2fbf290a82393d1c7dfcd5 not found: ID does not exist" containerID="929ac8585ac4e7d4661e9d2ea4c9eff0cd71ec823f2fbf290a82393d1c7dfcd5"
Oct 30 13:22:24.590325 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:24.590249    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"929ac8585ac4e7d4661e9d2ea4c9eff0cd71ec823f2fbf290a82393d1c7dfcd5"} err="failed to get container status \"929ac8585ac4e7d4661e9d2ea4c9eff0cd71ec823f2fbf290a82393d1c7dfcd5\": rpc error: code = NotFound desc = could not find container \"929ac8585ac4e7d4661e9d2ea4c9eff0cd71ec823f2fbf290a82393d1c7dfcd5\": container with ID starting with 929ac8585ac4e7d4661e9d2ea4c9eff0cd71ec823f2fbf290a82393d1c7dfcd5 not found: ID does not exist"
Oct 30 13:22:24.634340 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:24.634305    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-6b7b93a3/e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5"]
Oct 30 13:22:24.638380 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:24.638349    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-6b7b93a3/e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5"]
Oct 30 13:22:24.930763 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:24.930719    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-6b7b93a3/e2e-lifecycle-model-6b7b93a3-kserve-59cc97dc5c-ghtn5" podUID="81a50a32-4858-4cfc-94d0-754c8e89f6a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:22:25.001011 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:25.000976    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81a50a32-4858-4cfc-94d0-754c8e89f6a4" path="/var/lib/kubelet/pods/81a50a32-4858-4cfc-94d0-754c8e89f6a4/volumes"
Oct 30 13:22:44.238823 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:44.238779    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95"]
Oct 30 13:22:44.239835 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:44.239817    5093 cpu_manager.go:414] "containerMap: removing container" podUID="81a50a32-4858-4cfc-94d0-754c8e89f6a4" containerName="main"
Oct 30 13:22:44.239930 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:44.239837    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81a50a32-4858-4cfc-94d0-754c8e89f6a4" containerName="main"
Oct 30 13:22:44.239996 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:44.239982    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="81a50a32-4858-4cfc-94d0-754c8e89f6a4" containerName="main"
Oct 30 13:22:44.288435 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:44.288388    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95"]
Oct 30 13:22:44.288620 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:44.288535    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95"
Oct 30 13:22:44.308737 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:44.308693    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-8fb1a8-openshift-default-dockercfg-qqq4w\""
Oct 30 13:22:44.357696 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:44.353306    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3c2202-98944d8f9-vxvvs"]
Oct 30 13:22:44.357696 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:44.353583    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c2202-98944d8f9-vxvvs" podUID="3330a40e-0641-4286-a850-f37c0b25110d" containerName="maas-api" containerID="cri-o://076cca7837bb30d2da69f3f35ee8eeb3ee8ca253ba90ec04f6f82999128e229b" gracePeriod=30
Oct 30 13:22:44.407281 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:44.407241    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca419c67-141b-4a04-8ee6-95301302c378-istiod-ca-cert\") pod \"e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95\" (UID: \"ca419c67-141b-4a04-8ee6-95301302c378\") " pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95"
Oct 30 13:22:44.407436 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:44.407403    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca419c67-141b-4a04-8ee6-95301302c378-credential-socket\") pod \"e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95\" (UID: \"ca419c67-141b-4a04-8ee6-95301302c378\") " pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95"
Oct 30 13:22:44.407536 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:44.407516    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca419c67-141b-4a04-8ee6-95301302c378-istio-envoy\") pod \"e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95\" (UID: \"ca419c67-141b-4a04-8ee6-95301302c378\") " pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95"
Oct 30 13:22:44.407597 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:44.407581    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca419c67-141b-4a04-8ee6-95301302c378-istio-podinfo\") pod \"e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95\" (UID: \"ca419c67-141b-4a04-8ee6-95301302c378\") " pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95"
Oct 30 13:22:44.407642 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:44.407608    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca419c67-141b-4a04-8ee6-95301302c378-istio-token\") pod \"e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95\" (UID: \"ca419c67-141b-4a04-8ee6-95301302c378\") " pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95"
Oct 30 13:22:44.407681 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:44.407644    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca419c67-141b-4a04-8ee6-95301302c378-workload-certs\") pod \"e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95\" (UID: \"ca419c67-141b-4a04-8ee6-95301302c378\") " pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95"
Oct 30 13:22:44.407681 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:44.407670    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca419c67-141b-4a04-8ee6-95301302c378-istio-data\") pod \"e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95\" (UID: \"ca419c67-141b-4a04-8ee6-95301302c378\") " pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95"
Oct 30 13:22:44.407759 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:44.407736    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5xk4\" (UniqueName: \"kubernetes.io/projected/ca419c67-141b-4a04-8ee6-95301302c378-kube-api-access-c5xk4\") pod \"e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95\" (UID: \"ca419c67-141b-4a04-8ee6-95301302c378\") " pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95"
Oct 30 13:22:44.407809 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:44.407787    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca419c67-141b-4a04-8ee6-95301302c378-workload-socket\") pod \"e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95\" (UID: \"ca419c67-141b-4a04-8ee6-95301302c378\") " pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95"
Oct 30 13:22:44.509800 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:44.509705    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5xk4\" (UniqueName: \"kubernetes.io/projected/ca419c67-141b-4a04-8ee6-95301302c378-kube-api-access-c5xk4\") pod \"e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95\" (UID: \"ca419c67-141b-4a04-8ee6-95301302c378\") " pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95"
Oct 30 13:22:44.509800 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:44.509746    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca419c67-141b-4a04-8ee6-95301302c378-workload-socket\") pod \"e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95\" (UID: \"ca419c67-141b-4a04-8ee6-95301302c378\") " pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95"
Oct 30 13:22:44.510041 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:44.509854    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca419c67-141b-4a04-8ee6-95301302c378-istiod-ca-cert\") pod \"e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95\" (UID: \"ca419c67-141b-4a04-8ee6-95301302c378\") " pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95"
Oct 30 13:22:44.510041 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:44.509957    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca419c67-141b-4a04-8ee6-95301302c378-credential-socket\") pod \"e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95\" (UID: \"ca419c67-141b-4a04-8ee6-95301302c378\") " pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95"
Oct 30 13:22:44.510230 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:44.510191    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca419c67-141b-4a04-8ee6-95301302c378-istio-envoy\") pod \"e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95\" (UID: \"ca419c67-141b-4a04-8ee6-95301302c378\") " pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95"
Oct 30 13:22:44.510293 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:44.510253    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca419c67-141b-4a04-8ee6-95301302c378-istio-podinfo\") pod \"e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95\" (UID: \"ca419c67-141b-4a04-8ee6-95301302c378\") " pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95"
Oct 30 13:22:44.510293 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:44.510268    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca419c67-141b-4a04-8ee6-95301302c378-istio-token\") pod \"e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95\" (UID: \"ca419c67-141b-4a04-8ee6-95301302c378\") " pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95"
Oct 30 13:22:44.510730 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:44.510293    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca419c67-141b-4a04-8ee6-95301302c378-workload-certs\") pod \"e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95\" (UID: \"ca419c67-141b-4a04-8ee6-95301302c378\") " pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95"
Oct 30 13:22:44.510834 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:44.510749    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca419c67-141b-4a04-8ee6-95301302c378-istio-data\") pod \"e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95\" (UID: \"ca419c67-141b-4a04-8ee6-95301302c378\") " pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95"
Oct 30 13:22:44.511768 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:44.511741    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca419c67-141b-4a04-8ee6-95301302c378-istio-data\") pod \"e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95\" (UID: \"ca419c67-141b-4a04-8ee6-95301302c378\") " pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95"
Oct 30 13:22:44.512727 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:44.512350    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca419c67-141b-4a04-8ee6-95301302c378-workload-socket\") pod \"e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95\" (UID: \"ca419c67-141b-4a04-8ee6-95301302c378\") " pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95"
Oct 30 13:22:44.512727 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:44.512429    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca419c67-141b-4a04-8ee6-95301302c378-istiod-ca-cert\") pod \"e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95\" (UID: \"ca419c67-141b-4a04-8ee6-95301302c378\") " pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95"
Oct 30 13:22:44.512727 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:44.512649    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca419c67-141b-4a04-8ee6-95301302c378-credential-socket\") pod \"e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95\" (UID: \"ca419c67-141b-4a04-8ee6-95301302c378\") " pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95"
Oct 30 13:22:44.513044 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:44.513001    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca419c67-141b-4a04-8ee6-95301302c378-workload-certs\") pod \"e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95\" (UID: \"ca419c67-141b-4a04-8ee6-95301302c378\") " pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95"
Oct 30 13:22:44.522421 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:44.522325    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca419c67-141b-4a04-8ee6-95301302c378-istio-podinfo\") pod \"e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95\" (UID: \"ca419c67-141b-4a04-8ee6-95301302c378\") " pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95"
Oct 30 13:22:44.524854 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:44.524792    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca419c67-141b-4a04-8ee6-95301302c378-istio-envoy\") pod \"e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95\" (UID: \"ca419c67-141b-4a04-8ee6-95301302c378\") " pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95"
Oct 30 13:22:44.533183 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:44.533099    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca419c67-141b-4a04-8ee6-95301302c378-istio-token\") pod \"e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95\" (UID: \"ca419c67-141b-4a04-8ee6-95301302c378\") " pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95"
Oct 30 13:22:44.564706 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:44.564656    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5xk4\" (UniqueName: \"kubernetes.io/projected/ca419c67-141b-4a04-8ee6-95301302c378-kube-api-access-c5xk4\") pod \"e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95\" (UID: \"ca419c67-141b-4a04-8ee6-95301302c378\") " pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95"
Oct 30 13:22:44.639598 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:44.639515    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95"
Oct 30 13:22:44.760916 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:44.760827    5093 generic.go:356] "Generic (PLEG): container finished" podID="3330a40e-0641-4286-a850-f37c0b25110d" containerID="076cca7837bb30d2da69f3f35ee8eeb3ee8ca253ba90ec04f6f82999128e229b" exitCode=0
Oct 30 13:22:44.761088 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:44.761038    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c2202-98944d8f9-vxvvs" event={"ID":"3330a40e-0641-4286-a850-f37c0b25110d","Type":"ContainerDied","Data":"076cca7837bb30d2da69f3f35ee8eeb3ee8ca253ba90ec04f6f82999128e229b"}
Oct 30 13:22:44.761210 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:44.761141    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c2202-98944d8f9-vxvvs" event={"ID":"3330a40e-0641-4286-a850-f37c0b25110d","Type":"ContainerDied","Data":"5762e89b6d82394a14324fd74f997f91f5a3fc625ece795e34c295bccc4198c7"}
Oct 30 13:22:44.761210 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:44.761170    5093 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5762e89b6d82394a14324fd74f997f91f5a3fc625ece795e34c295bccc4198c7"
Oct 30 13:22:44.772956 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:44.772591    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c2202-98944d8f9-vxvvs"
Oct 30 13:22:44.950154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:44.938643    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3330a40e-0641-4286-a850-f37c0b25110d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3330a40e-0641-4286-a850-f37c0b25110d-metrics-certs\") pod \"3330a40e-0641-4286-a850-f37c0b25110d\" (UID: \"3330a40e-0641-4286-a850-f37c0b25110d\") "
Oct 30 13:22:44.950154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:44.938807    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3330a40e-0641-4286-a850-f37c0b25110d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3330a40e-0641-4286-a850-f37c0b25110d-openshift-service-ca\") pod \"3330a40e-0641-4286-a850-f37c0b25110d\" (UID: \"3330a40e-0641-4286-a850-f37c0b25110d\") "
Oct 30 13:22:44.950154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:44.938895    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3330a40e-0641-4286-a850-f37c0b25110d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3330a40e-0641-4286-a850-f37c0b25110d-maas-api-tls\") pod \"3330a40e-0641-4286-a850-f37c0b25110d\" (UID: \"3330a40e-0641-4286-a850-f37c0b25110d\") "
Oct 30 13:22:44.950154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:44.938950    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3330a40e-0641-4286-a850-f37c0b25110d-kube-api-access-8m2ch\" (UniqueName: \"kubernetes.io/projected/3330a40e-0641-4286-a850-f37c0b25110d-kube-api-access-8m2ch\") pod \"3330a40e-0641-4286-a850-f37c0b25110d\" (UID: \"3330a40e-0641-4286-a850-f37c0b25110d\") "
Oct 30 13:22:44.950154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:44.940259    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3330a40e-0641-4286-a850-f37c0b25110d-openshift-service-ca" pod "3330a40e-0641-4286-a850-f37c0b25110d" (UID: "3330a40e-0641-4286-a850-f37c0b25110d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:22:44.950154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:44.941940    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3330a40e-0641-4286-a850-f37c0b25110d-kube-api-access-8m2ch" pod "3330a40e-0641-4286-a850-f37c0b25110d" (UID: "3330a40e-0641-4286-a850-f37c0b25110d"). InnerVolumeSpecName "kube-api-access-8m2ch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:22:44.958184 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:44.958073    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3330a40e-0641-4286-a850-f37c0b25110d-metrics-certs" pod "3330a40e-0641-4286-a850-f37c0b25110d" (UID: "3330a40e-0641-4286-a850-f37c0b25110d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:22:44.962641 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:44.962607    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3330a40e-0641-4286-a850-f37c0b25110d-maas-api-tls" pod "3330a40e-0641-4286-a850-f37c0b25110d" (UID: "3330a40e-0641-4286-a850-f37c0b25110d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:22:45.041343 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:45.041307    5093 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3330a40e-0641-4286-a850-f37c0b25110d-metrics-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:22:45.041343 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:45.041345    5093 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3330a40e-0641-4286-a850-f37c0b25110d-openshift-service-ca\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:22:45.041569 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:45.041359    5093 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3330a40e-0641-4286-a850-f37c0b25110d-maas-api-tls\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:22:45.041569 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:45.041371    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m2ch\" (UniqueName: \"kubernetes.io/projected/3330a40e-0641-4286-a850-f37c0b25110d-kube-api-access-8m2ch\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:22:45.068877 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:45.068846    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95"]
Oct 30 13:22:45.069995 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:22:45.069962    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca419c67_141b_4a04_8ee6_95301302c378.slice/crio-553d73e3eb865d3a4620bdf58b59f4a09a89cfc058b57c39c823ddc0b42d859b WatchSource:0}: Error finding container 553d73e3eb865d3a4620bdf58b59f4a09a89cfc058b57c39c823ddc0b42d859b: Status 404 returned error can't find the container with id 553d73e3eb865d3a4620bdf58b59f4a09a89cfc058b57c39c823ddc0b42d859b
Oct 30 13:22:45.073002 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:45.072962    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:22:45.073109 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:45.073054    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:22:45.073109 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:45.073100    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:22:45.770334 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:45.770293    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95" event={"ID":"ca419c67-141b-4a04-8ee6-95301302c378","Type":"ContainerStarted","Data":"5b70a8c379098929a76f02957700146aa58a065c0ca6ce32a4f4e7d72c53427e"}
Oct 30 13:22:45.770826 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:45.770352    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95" event={"ID":"ca419c67-141b-4a04-8ee6-95301302c378","Type":"ContainerStarted","Data":"553d73e3eb865d3a4620bdf58b59f4a09a89cfc058b57c39c823ddc0b42d859b"}
Oct 30 13:22:45.770826 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:45.770391    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c2202-98944d8f9-vxvvs"
Oct 30 13:22:45.833448 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:45.832160    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95" podStartSLOduration=1.832139207 podStartE2EDuration="1.832139207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:22:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:22:45.813971187 +0000 UTC m=+1184.961238425" watchObservedRunningTime="2026-10-30 13:22:45.832139207 +0000 UTC m=+1184.979406443"
Oct 30 13:22:45.858225 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:45.858181    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3c2202-98944d8f9-vxvvs"]
Oct 30 13:22:45.863482 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:45.863449    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3c2202-98944d8f9-vxvvs"]
Oct 30 13:22:46.640398 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:46.640362    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95"
Oct 30 13:22:46.646532 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:46.646504    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95"
Oct 30 13:22:46.783049 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:46.783014    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95"
Oct 30 13:22:46.784081 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:46.784056    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95"
Oct 30 13:22:47.001454 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:47.001412    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3330a40e-0641-4286-a850-f37c0b25110d" path="/var/lib/kubelet/pods/3330a40e-0641-4286-a850-f37c0b25110d/volumes"
Oct 30 13:22:51.167837 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:22:51.167788    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:22:54.553653 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:54.553608    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf"]
Oct 30 13:22:54.555354 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:54.555314    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf" podUID="528a3a61-adc9-45fd-aefd-1cc67d8cf716" containerName="istio-proxy" containerID="cri-o://58e77bdcd8cfff405a7118e261acf4340e27edd9f57e3fa81bfb663fa4a95ac7" gracePeriod=30
Oct 30 13:22:55.842809 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:55.842770    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw"]
Oct 30 13:22:55.844457 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:55.844425    5093 cpu_manager.go:414] "containerMap: removing container" podUID="3330a40e-0641-4286-a850-f37c0b25110d" containerName="maas-api"
Oct 30 13:22:55.844457 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:55.844457    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3330a40e-0641-4286-a850-f37c0b25110d" containerName="maas-api"
Oct 30 13:22:55.844685 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:55.844649    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="3330a40e-0641-4286-a850-f37c0b25110d" containerName="maas-api"
Oct 30 13:22:55.850123 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:55.850096    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw"
Oct 30 13:22:55.873574 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:55.873543    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-c9e946-openshift-default-dockercfg-86xnj\""
Oct 30 13:22:55.883158 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:55.883110    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw"]
Oct 30 13:22:55.949318 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:55.949279    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20970fe5-df45-4db7-8d63-6c224f8bf7bb-istio-podinfo\") pod \"e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw\" (UID: \"20970fe5-df45-4db7-8d63-6c224f8bf7bb\") " pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw"
Oct 30 13:22:55.949662 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:55.949635    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20970fe5-df45-4db7-8d63-6c224f8bf7bb-istio-envoy\") pod \"e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw\" (UID: \"20970fe5-df45-4db7-8d63-6c224f8bf7bb\") " pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw"
Oct 30 13:22:55.949803 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:55.949701    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr5fc\" (UniqueName: \"kubernetes.io/projected/20970fe5-df45-4db7-8d63-6c224f8bf7bb-kube-api-access-sr5fc\") pod \"e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw\" (UID: \"20970fe5-df45-4db7-8d63-6c224f8bf7bb\") " pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw"
Oct 30 13:22:55.949803 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:55.949737    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20970fe5-df45-4db7-8d63-6c224f8bf7bb-istiod-ca-cert\") pod \"e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw\" (UID: \"20970fe5-df45-4db7-8d63-6c224f8bf7bb\") " pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw"
Oct 30 13:22:55.949803 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:55.949800    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20970fe5-df45-4db7-8d63-6c224f8bf7bb-workload-socket\") pod \"e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw\" (UID: \"20970fe5-df45-4db7-8d63-6c224f8bf7bb\") " pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw"
Oct 30 13:22:55.949984 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:55.949923    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20970fe5-df45-4db7-8d63-6c224f8bf7bb-workload-certs\") pod \"e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw\" (UID: \"20970fe5-df45-4db7-8d63-6c224f8bf7bb\") " pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw"
Oct 30 13:22:55.950049 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:55.949983    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20970fe5-df45-4db7-8d63-6c224f8bf7bb-istio-token\") pod \"e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw\" (UID: \"20970fe5-df45-4db7-8d63-6c224f8bf7bb\") " pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw"
Oct 30 13:22:55.950049 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:55.950034    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20970fe5-df45-4db7-8d63-6c224f8bf7bb-credential-socket\") pod \"e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw\" (UID: \"20970fe5-df45-4db7-8d63-6c224f8bf7bb\") " pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw"
Oct 30 13:22:55.950190 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:55.950100    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20970fe5-df45-4db7-8d63-6c224f8bf7bb-istio-data\") pod \"e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw\" (UID: \"20970fe5-df45-4db7-8d63-6c224f8bf7bb\") " pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw"
Oct 30 13:22:56.055296 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:56.055258    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20970fe5-df45-4db7-8d63-6c224f8bf7bb-istio-podinfo\") pod \"e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw\" (UID: \"20970fe5-df45-4db7-8d63-6c224f8bf7bb\") " pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw"
Oct 30 13:22:56.055625 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:56.055601    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20970fe5-df45-4db7-8d63-6c224f8bf7bb-istio-envoy\") pod \"e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw\" (UID: \"20970fe5-df45-4db7-8d63-6c224f8bf7bb\") " pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw"
Oct 30 13:22:56.055764 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:56.055750    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr5fc\" (UniqueName: \"kubernetes.io/projected/20970fe5-df45-4db7-8d63-6c224f8bf7bb-kube-api-access-sr5fc\") pod \"e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw\" (UID: \"20970fe5-df45-4db7-8d63-6c224f8bf7bb\") " pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw"
Oct 30 13:22:56.055866 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:56.055855    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20970fe5-df45-4db7-8d63-6c224f8bf7bb-istiod-ca-cert\") pod \"e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw\" (UID: \"20970fe5-df45-4db7-8d63-6c224f8bf7bb\") " pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw"
Oct 30 13:22:56.055993 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:56.055982    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20970fe5-df45-4db7-8d63-6c224f8bf7bb-workload-socket\") pod \"e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw\" (UID: \"20970fe5-df45-4db7-8d63-6c224f8bf7bb\") " pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw"
Oct 30 13:22:56.056145 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:56.056118    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20970fe5-df45-4db7-8d63-6c224f8bf7bb-workload-certs\") pod \"e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw\" (UID: \"20970fe5-df45-4db7-8d63-6c224f8bf7bb\") " pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw"
Oct 30 13:22:56.056301 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:56.056288    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20970fe5-df45-4db7-8d63-6c224f8bf7bb-istio-token\") pod \"e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw\" (UID: \"20970fe5-df45-4db7-8d63-6c224f8bf7bb\") " pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw"
Oct 30 13:22:56.057294 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:56.057265    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20970fe5-df45-4db7-8d63-6c224f8bf7bb-credential-socket\") pod \"e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw\" (UID: \"20970fe5-df45-4db7-8d63-6c224f8bf7bb\") " pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw"
Oct 30 13:22:56.057493 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:56.057478    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20970fe5-df45-4db7-8d63-6c224f8bf7bb-istio-data\") pod \"e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw\" (UID: \"20970fe5-df45-4db7-8d63-6c224f8bf7bb\") " pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw"
Oct 30 13:22:56.064341 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:56.057638    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20970fe5-df45-4db7-8d63-6c224f8bf7bb-workload-certs\") pod \"e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw\" (UID: \"20970fe5-df45-4db7-8d63-6c224f8bf7bb\") " pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw"
Oct 30 13:22:56.064487 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:56.056621    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20970fe5-df45-4db7-8d63-6c224f8bf7bb-workload-socket\") pod \"e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw\" (UID: \"20970fe5-df45-4db7-8d63-6c224f8bf7bb\") " pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw"
Oct 30 13:22:56.064720 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:56.064689    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20970fe5-df45-4db7-8d63-6c224f8bf7bb-credential-socket\") pod \"e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw\" (UID: \"20970fe5-df45-4db7-8d63-6c224f8bf7bb\") " pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw"
Oct 30 13:22:56.064826 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:56.057367    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20970fe5-df45-4db7-8d63-6c224f8bf7bb-istiod-ca-cert\") pod \"e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw\" (UID: \"20970fe5-df45-4db7-8d63-6c224f8bf7bb\") " pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw"
Oct 30 13:22:56.065011 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:56.064986    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20970fe5-df45-4db7-8d63-6c224f8bf7bb-istio-data\") pod \"e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw\" (UID: \"20970fe5-df45-4db7-8d63-6c224f8bf7bb\") " pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw"
Oct 30 13:22:56.122210 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:56.122116    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20970fe5-df45-4db7-8d63-6c224f8bf7bb-istio-podinfo\") pod \"e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw\" (UID: \"20970fe5-df45-4db7-8d63-6c224f8bf7bb\") " pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw"
Oct 30 13:22:56.132809 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:56.131883    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr5fc\" (UniqueName: \"kubernetes.io/projected/20970fe5-df45-4db7-8d63-6c224f8bf7bb-kube-api-access-sr5fc\") pod \"e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw\" (UID: \"20970fe5-df45-4db7-8d63-6c224f8bf7bb\") " pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw"
Oct 30 13:22:56.134324 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:56.133337    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20970fe5-df45-4db7-8d63-6c224f8bf7bb-istio-envoy\") pod \"e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw\" (UID: \"20970fe5-df45-4db7-8d63-6c224f8bf7bb\") " pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw"
Oct 30 13:22:56.197157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:56.178080    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20970fe5-df45-4db7-8d63-6c224f8bf7bb-istio-token\") pod \"e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw\" (UID: \"20970fe5-df45-4db7-8d63-6c224f8bf7bb\") " pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw"
Oct 30 13:22:56.219359 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:56.219317    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw"
Oct 30 13:22:56.581181 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:22:56.581144    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20970fe5_df45_4db7_8d63_6c224f8bf7bb.slice/crio-8aaf968a5148c0da0f55a7efdbe086278e9906ceed6474b8c1068382443be1cb WatchSource:0}: Error finding container 8aaf968a5148c0da0f55a7efdbe086278e9906ceed6474b8c1068382443be1cb: Status 404 returned error can't find the container with id 8aaf968a5148c0da0f55a7efdbe086278e9906ceed6474b8c1068382443be1cb
Oct 30 13:22:56.582524 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:56.582484    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw"]
Oct 30 13:22:56.584191 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:56.584158    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:22:56.584293 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:56.584225    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:22:56.584293 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:56.584253    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:22:56.904641 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:56.904544    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw" event={"ID":"20970fe5-df45-4db7-8d63-6c224f8bf7bb","Type":"ContainerStarted","Data":"c1c6ab2defcaedba8959f85583421e3c8c6de7d18786ba04c9a02f54c7de30ed"}
Oct 30 13:22:56.904641 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:56.904602    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw" event={"ID":"20970fe5-df45-4db7-8d63-6c224f8bf7bb","Type":"ContainerStarted","Data":"8aaf968a5148c0da0f55a7efdbe086278e9906ceed6474b8c1068382443be1cb"}
Oct 30 13:22:56.947619 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:56.947561    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw" podStartSLOduration=1.947547307 podStartE2EDuration="1.947547307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:22:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:22:56.946526756 +0000 UTC m=+1196.093794011" watchObservedRunningTime="2026-10-30 13:22:56.947547307 +0000 UTC m=+1196.094814548"
Oct 30 13:22:57.220483 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:57.220443    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw"
Oct 30 13:22:57.226623 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:57.226596    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw"
Oct 30 13:22:57.912793 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:57.912759    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw"
Oct 30 13:22:57.913958 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:57.913936    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw"
Oct 30 13:22:59.863720 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:59.863685    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf"
Oct 30 13:22:59.940359 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:59.940323    5093 generic.go:356] "Generic (PLEG): container finished" podID="528a3a61-adc9-45fd-aefd-1cc67d8cf716" containerID="58e77bdcd8cfff405a7118e261acf4340e27edd9f57e3fa81bfb663fa4a95ac7" exitCode=0
Oct 30 13:22:59.940542 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:59.940435    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf"
Oct 30 13:22:59.940542 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:59.940463    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf" event={"ID":"528a3a61-adc9-45fd-aefd-1cc67d8cf716","Type":"ContainerDied","Data":"58e77bdcd8cfff405a7118e261acf4340e27edd9f57e3fa81bfb663fa4a95ac7"}
Oct 30 13:22:59.940542 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:59.940529    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf" event={"ID":"528a3a61-adc9-45fd-aefd-1cc67d8cf716","Type":"ContainerDied","Data":"32b7fc50db424eba404bb56363cc1f0b5cfd4bb1edc5f00d40aac94e4fd8b3a1"}
Oct 30 13:22:59.940719 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:59.940559    5093 scope.go:122] "RemoveContainer" containerID="58e77bdcd8cfff405a7118e261acf4340e27edd9f57e3fa81bfb663fa4a95ac7"
Oct 30 13:22:59.943570 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:59.943543    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/528a3a61-adc9-45fd-aefd-1cc67d8cf716-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/528a3a61-adc9-45fd-aefd-1cc67d8cf716-istiod-ca-cert\") pod \"528a3a61-adc9-45fd-aefd-1cc67d8cf716\" (UID: \"528a3a61-adc9-45fd-aefd-1cc67d8cf716\") "
Oct 30 13:22:59.943716 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:59.943603    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/528a3a61-adc9-45fd-aefd-1cc67d8cf716-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/528a3a61-adc9-45fd-aefd-1cc67d8cf716-istio-data\") pod \"528a3a61-adc9-45fd-aefd-1cc67d8cf716\" (UID: \"528a3a61-adc9-45fd-aefd-1cc67d8cf716\") "
Oct 30 13:22:59.943716 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:59.943653    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/528a3a61-adc9-45fd-aefd-1cc67d8cf716-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/528a3a61-adc9-45fd-aefd-1cc67d8cf716-istio-envoy\") pod \"528a3a61-adc9-45fd-aefd-1cc67d8cf716\" (UID: \"528a3a61-adc9-45fd-aefd-1cc67d8cf716\") "
Oct 30 13:22:59.943716 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:59.943679    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/528a3a61-adc9-45fd-aefd-1cc67d8cf716-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/528a3a61-adc9-45fd-aefd-1cc67d8cf716-credential-socket\") pod \"528a3a61-adc9-45fd-aefd-1cc67d8cf716\" (UID: \"528a3a61-adc9-45fd-aefd-1cc67d8cf716\") "
Oct 30 13:22:59.943897 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:59.943721    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/528a3a61-adc9-45fd-aefd-1cc67d8cf716-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/528a3a61-adc9-45fd-aefd-1cc67d8cf716-istio-podinfo\") pod \"528a3a61-adc9-45fd-aefd-1cc67d8cf716\" (UID: \"528a3a61-adc9-45fd-aefd-1cc67d8cf716\") "
Oct 30 13:22:59.943897 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:59.943859    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/528a3a61-adc9-45fd-aefd-1cc67d8cf716-kube-api-access-p7kv9\" (UniqueName: \"kubernetes.io/projected/528a3a61-adc9-45fd-aefd-1cc67d8cf716-kube-api-access-p7kv9\") pod \"528a3a61-adc9-45fd-aefd-1cc67d8cf716\" (UID: \"528a3a61-adc9-45fd-aefd-1cc67d8cf716\") "
Oct 30 13:22:59.943995 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:59.943902    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/528a3a61-adc9-45fd-aefd-1cc67d8cf716-istio-data" pod "528a3a61-adc9-45fd-aefd-1cc67d8cf716" (UID: "528a3a61-adc9-45fd-aefd-1cc67d8cf716"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:22:59.943995 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:59.943945    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/528a3a61-adc9-45fd-aefd-1cc67d8cf716-istiod-ca-cert" pod "528a3a61-adc9-45fd-aefd-1cc67d8cf716" (UID: "528a3a61-adc9-45fd-aefd-1cc67d8cf716"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:22:59.943995 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:59.943982    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/528a3a61-adc9-45fd-aefd-1cc67d8cf716-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/528a3a61-adc9-45fd-aefd-1cc67d8cf716-workload-socket\") pod \"528a3a61-adc9-45fd-aefd-1cc67d8cf716\" (UID: \"528a3a61-adc9-45fd-aefd-1cc67d8cf716\") "
Oct 30 13:22:59.944202 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:59.944104    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/528a3a61-adc9-45fd-aefd-1cc67d8cf716-credential-socket" pod "528a3a61-adc9-45fd-aefd-1cc67d8cf716" (UID: "528a3a61-adc9-45fd-aefd-1cc67d8cf716"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:22:59.944202 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:59.944171    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/528a3a61-adc9-45fd-aefd-1cc67d8cf716-istio-token\" (UniqueName: \"kubernetes.io/projected/528a3a61-adc9-45fd-aefd-1cc67d8cf716-istio-token\") pod \"528a3a61-adc9-45fd-aefd-1cc67d8cf716\" (UID: \"528a3a61-adc9-45fd-aefd-1cc67d8cf716\") "
Oct 30 13:22:59.944317 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:59.944285    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/528a3a61-adc9-45fd-aefd-1cc67d8cf716-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/528a3a61-adc9-45fd-aefd-1cc67d8cf716-workload-certs\") pod \"528a3a61-adc9-45fd-aefd-1cc67d8cf716\" (UID: \"528a3a61-adc9-45fd-aefd-1cc67d8cf716\") "
Oct 30 13:22:59.944759 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:59.944733    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/528a3a61-adc9-45fd-aefd-1cc67d8cf716-workload-socket" pod "528a3a61-adc9-45fd-aefd-1cc67d8cf716" (UID: "528a3a61-adc9-45fd-aefd-1cc67d8cf716"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:22:59.944993 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:59.944969    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/528a3a61-adc9-45fd-aefd-1cc67d8cf716-workload-certs" pod "528a3a61-adc9-45fd-aefd-1cc67d8cf716" (UID: "528a3a61-adc9-45fd-aefd-1cc67d8cf716"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:22:59.945072 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:59.945036    5093 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/528a3a61-adc9-45fd-aefd-1cc67d8cf716-istiod-ca-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:22:59.945072 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:59.945051    5093 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/528a3a61-adc9-45fd-aefd-1cc67d8cf716-istio-data\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:22:59.945072 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:59.945060    5093 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/528a3a61-adc9-45fd-aefd-1cc67d8cf716-credential-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:22:59.945072 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:59.945069    5093 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/528a3a61-adc9-45fd-aefd-1cc67d8cf716-workload-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:22:59.947461 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:59.947434    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/528a3a61-adc9-45fd-aefd-1cc67d8cf716-istio-token" pod "528a3a61-adc9-45fd-aefd-1cc67d8cf716" (UID: "528a3a61-adc9-45fd-aefd-1cc67d8cf716"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:22:59.947548 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:59.947477    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/528a3a61-adc9-45fd-aefd-1cc67d8cf716-kube-api-access-p7kv9" pod "528a3a61-adc9-45fd-aefd-1cc67d8cf716" (UID: "528a3a61-adc9-45fd-aefd-1cc67d8cf716"). InnerVolumeSpecName "kube-api-access-p7kv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:22:59.947702 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:59.947678    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/528a3a61-adc9-45fd-aefd-1cc67d8cf716-istio-envoy" pod "528a3a61-adc9-45fd-aefd-1cc67d8cf716" (UID: "528a3a61-adc9-45fd-aefd-1cc67d8cf716"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:22:59.955022 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:59.954988    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/528a3a61-adc9-45fd-aefd-1cc67d8cf716-istio-podinfo" pod "528a3a61-adc9-45fd-aefd-1cc67d8cf716" (UID: "528a3a61-adc9-45fd-aefd-1cc67d8cf716"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:22:59.961765 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:59.961742    5093 scope.go:122] "RemoveContainer" containerID="c7944a7e350c9ad5a27bc5f7a09cabd57d9ad46fdb7ec8104d2a6c7186d462d4"
Oct 30 13:22:59.996108 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:59.996080    5093 scope.go:122] "RemoveContainer" containerID="58e77bdcd8cfff405a7118e261acf4340e27edd9f57e3fa81bfb663fa4a95ac7"
Oct 30 13:22:59.996468 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:22:59.996446    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58e77bdcd8cfff405a7118e261acf4340e27edd9f57e3fa81bfb663fa4a95ac7\": container with ID starting with 58e77bdcd8cfff405a7118e261acf4340e27edd9f57e3fa81bfb663fa4a95ac7 not found: ID does not exist" containerID="58e77bdcd8cfff405a7118e261acf4340e27edd9f57e3fa81bfb663fa4a95ac7"
Oct 30 13:22:59.996558 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:59.996482    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58e77bdcd8cfff405a7118e261acf4340e27edd9f57e3fa81bfb663fa4a95ac7"} err="failed to get container status \"58e77bdcd8cfff405a7118e261acf4340e27edd9f57e3fa81bfb663fa4a95ac7\": rpc error: code = NotFound desc = could not find container \"58e77bdcd8cfff405a7118e261acf4340e27edd9f57e3fa81bfb663fa4a95ac7\": container with ID starting with 58e77bdcd8cfff405a7118e261acf4340e27edd9f57e3fa81bfb663fa4a95ac7 not found: ID does not exist"
Oct 30 13:22:59.996558 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:59.996503    5093 scope.go:122] "RemoveContainer" containerID="c7944a7e350c9ad5a27bc5f7a09cabd57d9ad46fdb7ec8104d2a6c7186d462d4"
Oct 30 13:22:59.996810 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:22:59.996782    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7944a7e350c9ad5a27bc5f7a09cabd57d9ad46fdb7ec8104d2a6c7186d462d4\": container with ID starting with c7944a7e350c9ad5a27bc5f7a09cabd57d9ad46fdb7ec8104d2a6c7186d462d4 not found: ID does not exist" containerID="c7944a7e350c9ad5a27bc5f7a09cabd57d9ad46fdb7ec8104d2a6c7186d462d4"
Oct 30 13:22:59.996851 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:22:59.996809    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7944a7e350c9ad5a27bc5f7a09cabd57d9ad46fdb7ec8104d2a6c7186d462d4"} err="failed to get container status \"c7944a7e350c9ad5a27bc5f7a09cabd57d9ad46fdb7ec8104d2a6c7186d462d4\": rpc error: code = NotFound desc = could not find container \"c7944a7e350c9ad5a27bc5f7a09cabd57d9ad46fdb7ec8104d2a6c7186d462d4\": container with ID starting with c7944a7e350c9ad5a27bc5f7a09cabd57d9ad46fdb7ec8104d2a6c7186d462d4 not found: ID does not exist"
Oct 30 13:23:00.046298 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:00.046214    5093 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/528a3a61-adc9-45fd-aefd-1cc67d8cf716-istio-envoy\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:23:00.046298 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:00.046247    5093 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/528a3a61-adc9-45fd-aefd-1cc67d8cf716-istio-podinfo\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:23:00.046298 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:00.046257    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7kv9\" (UniqueName: \"kubernetes.io/projected/528a3a61-adc9-45fd-aefd-1cc67d8cf716-kube-api-access-p7kv9\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:23:00.046298 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:00.046266    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/528a3a61-adc9-45fd-aefd-1cc67d8cf716-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:23:00.046298 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:00.046276    5093 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/528a3a61-adc9-45fd-aefd-1cc67d8cf716-workload-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:23:00.292294 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:00.292257    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf"]
Oct 30 13:23:00.305255 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:00.305221    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6b7b93a3-openshift-default-589cd4c4b9-wcwqf"]
Oct 30 13:23:00.465541 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:00.465506    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf"]
Oct 30 13:23:00.466515 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:00.466489    5093 cpu_manager.go:414] "containerMap: removing container" podUID="528a3a61-adc9-45fd-aefd-1cc67d8cf716" containerName="istio-proxy"
Oct 30 13:23:00.466515 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:00.466512    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="528a3a61-adc9-45fd-aefd-1cc67d8cf716" containerName="istio-proxy"
Oct 30 13:23:00.466697 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:00.466556    5093 cpu_manager.go:414] "containerMap: removing container" podUID="528a3a61-adc9-45fd-aefd-1cc67d8cf716" containerName="istio-proxy"
Oct 30 13:23:00.466697 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:00.466564    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="528a3a61-adc9-45fd-aefd-1cc67d8cf716" containerName="istio-proxy"
Oct 30 13:23:00.466789 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:00.466721    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="528a3a61-adc9-45fd-aefd-1cc67d8cf716" containerName="istio-proxy"
Oct 30 13:23:00.466789 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:00.466740    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="528a3a61-adc9-45fd-aefd-1cc67d8cf716" containerName="istio-proxy"
Oct 30 13:23:00.474105 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:00.474082    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf"
Oct 30 13:23:00.485764 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:00.485516    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf"]
Oct 30 13:23:00.496478 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:00.496379    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-3a82db-openshift-default-dockercfg-lthwh\""
Oct 30 13:23:00.554960 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:00.554917    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58244ace-ebd4-4b98-929b-8396655992a6-istiod-ca-cert\") pod \"e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf\" (UID: \"58244ace-ebd4-4b98-929b-8396655992a6\") " pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf"
Oct 30 13:23:00.555201 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:00.554981    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58244ace-ebd4-4b98-929b-8396655992a6-credential-socket\") pod \"e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf\" (UID: \"58244ace-ebd4-4b98-929b-8396655992a6\") " pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf"
Oct 30 13:23:00.555201 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:00.555090    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58244ace-ebd4-4b98-929b-8396655992a6-workload-certs\") pod \"e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf\" (UID: \"58244ace-ebd4-4b98-929b-8396655992a6\") " pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf"
Oct 30 13:23:00.555349 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:00.555281    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58244ace-ebd4-4b98-929b-8396655992a6-istio-envoy\") pod \"e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf\" (UID: \"58244ace-ebd4-4b98-929b-8396655992a6\") " pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf"
Oct 30 13:23:00.555407 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:00.555343    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58244ace-ebd4-4b98-929b-8396655992a6-istio-podinfo\") pod \"e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf\" (UID: \"58244ace-ebd4-4b98-929b-8396655992a6\") " pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf"
Oct 30 13:23:00.555460 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:00.555413    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58244ace-ebd4-4b98-929b-8396655992a6-workload-socket\") pod \"e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf\" (UID: \"58244ace-ebd4-4b98-929b-8396655992a6\") " pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf"
Oct 30 13:23:00.555554 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:00.555528    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v42gh\" (UniqueName: \"kubernetes.io/projected/58244ace-ebd4-4b98-929b-8396655992a6-kube-api-access-v42gh\") pod \"e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf\" (UID: \"58244ace-ebd4-4b98-929b-8396655992a6\") " pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf"
Oct 30 13:23:00.555750 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:00.555727    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58244ace-ebd4-4b98-929b-8396655992a6-istio-data\") pod \"e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf\" (UID: \"58244ace-ebd4-4b98-929b-8396655992a6\") " pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf"
Oct 30 13:23:00.555804 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:00.555769    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58244ace-ebd4-4b98-929b-8396655992a6-istio-token\") pod \"e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf\" (UID: \"58244ace-ebd4-4b98-929b-8396655992a6\") " pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf"
Oct 30 13:23:00.670157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:00.665153    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58244ace-ebd4-4b98-929b-8396655992a6-istio-envoy\") pod \"e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf\" (UID: \"58244ace-ebd4-4b98-929b-8396655992a6\") " pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf"
Oct 30 13:23:00.670157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:00.665221    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58244ace-ebd4-4b98-929b-8396655992a6-istio-podinfo\") pod \"e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf\" (UID: \"58244ace-ebd4-4b98-929b-8396655992a6\") " pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf"
Oct 30 13:23:00.670157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:00.665273    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58244ace-ebd4-4b98-929b-8396655992a6-workload-socket\") pod \"e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf\" (UID: \"58244ace-ebd4-4b98-929b-8396655992a6\") " pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf"
Oct 30 13:23:00.670157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:00.665369    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v42gh\" (UniqueName: \"kubernetes.io/projected/58244ace-ebd4-4b98-929b-8396655992a6-kube-api-access-v42gh\") pod \"e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf\" (UID: \"58244ace-ebd4-4b98-929b-8396655992a6\") " pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf"
Oct 30 13:23:00.670157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:00.665466    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58244ace-ebd4-4b98-929b-8396655992a6-istio-data\") pod \"e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf\" (UID: \"58244ace-ebd4-4b98-929b-8396655992a6\") " pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf"
Oct 30 13:23:00.670157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:00.665508    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58244ace-ebd4-4b98-929b-8396655992a6-istio-token\") pod \"e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf\" (UID: \"58244ace-ebd4-4b98-929b-8396655992a6\") " pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf"
Oct 30 13:23:00.670157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:00.665589    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58244ace-ebd4-4b98-929b-8396655992a6-istiod-ca-cert\") pod \"e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf\" (UID: \"58244ace-ebd4-4b98-929b-8396655992a6\") " pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf"
Oct 30 13:23:00.670157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:00.665631    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58244ace-ebd4-4b98-929b-8396655992a6-credential-socket\") pod \"e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf\" (UID: \"58244ace-ebd4-4b98-929b-8396655992a6\") " pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf"
Oct 30 13:23:00.670157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:00.665677    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58244ace-ebd4-4b98-929b-8396655992a6-workload-certs\") pod \"e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf\" (UID: \"58244ace-ebd4-4b98-929b-8396655992a6\") " pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf"
Oct 30 13:23:00.670157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:00.666283    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58244ace-ebd4-4b98-929b-8396655992a6-workload-certs\") pod \"e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf\" (UID: \"58244ace-ebd4-4b98-929b-8396655992a6\") " pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf"
Oct 30 13:23:00.670157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:00.666567    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58244ace-ebd4-4b98-929b-8396655992a6-istio-data\") pod \"e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf\" (UID: \"58244ace-ebd4-4b98-929b-8396655992a6\") " pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf"
Oct 30 13:23:00.714237 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:00.707455    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58244ace-ebd4-4b98-929b-8396655992a6-workload-socket\") pod \"e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf\" (UID: \"58244ace-ebd4-4b98-929b-8396655992a6\") " pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf"
Oct 30 13:23:00.714237 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:00.708918    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58244ace-ebd4-4b98-929b-8396655992a6-istiod-ca-cert\") pod \"e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf\" (UID: \"58244ace-ebd4-4b98-929b-8396655992a6\") " pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf"
Oct 30 13:23:00.714237 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:00.712919    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58244ace-ebd4-4b98-929b-8396655992a6-credential-socket\") pod \"e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf\" (UID: \"58244ace-ebd4-4b98-929b-8396655992a6\") " pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf"
Oct 30 13:23:00.815257 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:00.791988    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v42gh\" (UniqueName: \"kubernetes.io/projected/58244ace-ebd4-4b98-929b-8396655992a6-kube-api-access-v42gh\") pod \"e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf\" (UID: \"58244ace-ebd4-4b98-929b-8396655992a6\") " pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf"
Oct 30 13:23:00.859832 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:00.859728    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58244ace-ebd4-4b98-929b-8396655992a6-istio-token\") pod \"e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf\" (UID: \"58244ace-ebd4-4b98-929b-8396655992a6\") " pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf"
Oct 30 13:23:00.884337 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:00.884300    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58244ace-ebd4-4b98-929b-8396655992a6-istio-podinfo\") pod \"e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf\" (UID: \"58244ace-ebd4-4b98-929b-8396655992a6\") " pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf"
Oct 30 13:23:00.885536 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:00.885509    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58244ace-ebd4-4b98-929b-8396655992a6-istio-envoy\") pod \"e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf\" (UID: \"58244ace-ebd4-4b98-929b-8396655992a6\") " pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf"
Oct 30 13:23:01.004153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:01.004097    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="528a3a61-adc9-45fd-aefd-1cc67d8cf716" path="/var/lib/kubelet/pods/528a3a61-adc9-45fd-aefd-1cc67d8cf716/volumes"
Oct 30 13:23:01.123660 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:01.123580    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-3a82db-openshift-default-dockercfg-lthwh\""
Oct 30 13:23:01.129815 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:01.129785    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf"
Oct 30 13:23:01.276085 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:01.276054    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jdj4_f93ee017-ab79-447c-90e0-5b85b71dba24/console-operator/1.log"
Oct 30 13:23:01.276273 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:01.276262    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jdj4_f93ee017-ab79-447c-90e0-5b85b71dba24/console-operator/1.log"
Oct 30 13:23:01.284553 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:01.284526    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tnm_3a675ac7-fbe1-449b-accf-bb0456ed0bd3/ovn-acl-logging/0.log"
Oct 30 13:23:01.284716 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:01.284526    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tnm_3a675ac7-fbe1-449b-accf-bb0456ed0bd3/ovn-acl-logging/0.log"
Oct 30 13:23:01.455814 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:01.455394    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf"]
Oct 30 13:23:01.457183 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:23:01.457113    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58244ace_ebd4_4b98_929b_8396655992a6.slice/crio-dfcec110b9514bafbd5c1ae12ca3984c12b1aad891e1db8b78b0d6b64fb01c6e WatchSource:0}: Error finding container dfcec110b9514bafbd5c1ae12ca3984c12b1aad891e1db8b78b0d6b64fb01c6e: Status 404 returned error can't find the container with id dfcec110b9514bafbd5c1ae12ca3984c12b1aad891e1db8b78b0d6b64fb01c6e
Oct 30 13:23:01.460883 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:01.460808    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:23:01.460992 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:01.460935    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:23:01.460992 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:01.460981    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:23:01.960777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:01.960739    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf" event={"ID":"58244ace-ebd4-4b98-929b-8396655992a6","Type":"ContainerStarted","Data":"bb40ce33d522380835ac6b8cf3867afdbf64bdf3052b03b320cfa389404fdf81"}
Oct 30 13:23:01.961274 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:01.960813    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf" event={"ID":"58244ace-ebd4-4b98-929b-8396655992a6","Type":"ContainerStarted","Data":"dfcec110b9514bafbd5c1ae12ca3984c12b1aad891e1db8b78b0d6b64fb01c6e"}
Oct 30 13:23:02.008528 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:02.008418    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf" podStartSLOduration=2.008397287 podStartE2EDuration="2.008397287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:23:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:23:02.00402372 +0000 UTC m=+1201.151290974" watchObservedRunningTime="2026-10-30 13:23:02.008397287 +0000 UTC m=+1201.155664529"
Oct 30 13:23:02.131188 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:02.131112    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf"
Oct 30 13:23:02.150891 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:02.145194    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf"
Oct 30 13:23:02.970942 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:02.970910    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf"
Oct 30 13:23:02.972411 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:02.972387    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf"
Oct 30 13:23:20.443697 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:20.443650    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c9e946-8f46bbc55-xmk9s"]
Oct 30 13:23:20.448266 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:20.448237    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9e946-8f46bbc55-xmk9s"
Oct 30 13:23:20.478557 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:20.478479    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-c9e946-serving-cert\""
Oct 30 13:23:20.480201 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:20.480166    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c9e946-8f46bbc55-xmk9s"]
Oct 30 13:23:20.535335 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:20.535299    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8662720a-f907-42b3-a58e-d37b854a9384-maas-api-tls\") pod \"maas-api-e2e-api-a-c9e946-8f46bbc55-xmk9s\" (UID: \"8662720a-f907-42b3-a58e-d37b854a9384\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9e946-8f46bbc55-xmk9s"
Oct 30 13:23:20.535561 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:20.535533    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pms8r\" (UniqueName: \"kubernetes.io/projected/8662720a-f907-42b3-a58e-d37b854a9384-kube-api-access-pms8r\") pod \"maas-api-e2e-api-a-c9e946-8f46bbc55-xmk9s\" (UID: \"8662720a-f907-42b3-a58e-d37b854a9384\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9e946-8f46bbc55-xmk9s"
Oct 30 13:23:20.535677 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:20.535596    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8662720a-f907-42b3-a58e-d37b854a9384-metrics-certs\") pod \"maas-api-e2e-api-a-c9e946-8f46bbc55-xmk9s\" (UID: \"8662720a-f907-42b3-a58e-d37b854a9384\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9e946-8f46bbc55-xmk9s"
Oct 30 13:23:20.535743 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:20.535681    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8662720a-f907-42b3-a58e-d37b854a9384-openshift-service-ca\") pod \"maas-api-e2e-api-a-c9e946-8f46bbc55-xmk9s\" (UID: \"8662720a-f907-42b3-a58e-d37b854a9384\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9e946-8f46bbc55-xmk9s"
Oct 30 13:23:20.637564 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:20.637465    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pms8r\" (UniqueName: \"kubernetes.io/projected/8662720a-f907-42b3-a58e-d37b854a9384-kube-api-access-pms8r\") pod \"maas-api-e2e-api-a-c9e946-8f46bbc55-xmk9s\" (UID: \"8662720a-f907-42b3-a58e-d37b854a9384\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9e946-8f46bbc55-xmk9s"
Oct 30 13:23:20.637564 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:20.637541    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8662720a-f907-42b3-a58e-d37b854a9384-metrics-certs\") pod \"maas-api-e2e-api-a-c9e946-8f46bbc55-xmk9s\" (UID: \"8662720a-f907-42b3-a58e-d37b854a9384\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9e946-8f46bbc55-xmk9s"
Oct 30 13:23:20.637932 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:20.637878    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8662720a-f907-42b3-a58e-d37b854a9384-openshift-service-ca\") pod \"maas-api-e2e-api-a-c9e946-8f46bbc55-xmk9s\" (UID: \"8662720a-f907-42b3-a58e-d37b854a9384\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9e946-8f46bbc55-xmk9s"
Oct 30 13:23:20.638378 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:20.638176    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8662720a-f907-42b3-a58e-d37b854a9384-maas-api-tls\") pod \"maas-api-e2e-api-a-c9e946-8f46bbc55-xmk9s\" (UID: \"8662720a-f907-42b3-a58e-d37b854a9384\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9e946-8f46bbc55-xmk9s"
Oct 30 13:23:20.639100 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:20.639078    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8662720a-f907-42b3-a58e-d37b854a9384-openshift-service-ca\") pod \"maas-api-e2e-api-a-c9e946-8f46bbc55-xmk9s\" (UID: \"8662720a-f907-42b3-a58e-d37b854a9384\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9e946-8f46bbc55-xmk9s"
Oct 30 13:23:20.640824 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:20.640800    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8662720a-f907-42b3-a58e-d37b854a9384-maas-api-tls\") pod \"maas-api-e2e-api-a-c9e946-8f46bbc55-xmk9s\" (UID: \"8662720a-f907-42b3-a58e-d37b854a9384\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9e946-8f46bbc55-xmk9s"
Oct 30 13:23:20.640900 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:20.640849    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8662720a-f907-42b3-a58e-d37b854a9384-metrics-certs\") pod \"maas-api-e2e-api-a-c9e946-8f46bbc55-xmk9s\" (UID: \"8662720a-f907-42b3-a58e-d37b854a9384\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9e946-8f46bbc55-xmk9s"
Oct 30 13:23:20.657961 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:20.657928    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pms8r\" (UniqueName: \"kubernetes.io/projected/8662720a-f907-42b3-a58e-d37b854a9384-kube-api-access-pms8r\") pod \"maas-api-e2e-api-a-c9e946-8f46bbc55-xmk9s\" (UID: \"8662720a-f907-42b3-a58e-d37b854a9384\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9e946-8f46bbc55-xmk9s"
Oct 30 13:23:20.765995 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:20.765878    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9e946-8f46bbc55-xmk9s"
Oct 30 13:23:21.086823 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:21.086794    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c9e946-8f46bbc55-xmk9s"]
Oct 30 13:23:21.087217 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:23:21.087186    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8662720a_f907_42b3_a58e_d37b854a9384.slice/crio-0cb13038ce245f08039edc691504653db141ca0b49451ef599c2b489cfcafa9a WatchSource:0}: Error finding container 0cb13038ce245f08039edc691504653db141ca0b49451ef599c2b489cfcafa9a: Status 404 returned error can't find the container with id 0cb13038ce245f08039edc691504653db141ca0b49451ef599c2b489cfcafa9a
Oct 30 13:23:21.169046 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:23:21.169007    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:23:21.179838 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:21.179805    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9e946-8f46bbc55-xmk9s" event={"ID":"8662720a-f907-42b3-a58e-d37b854a9384","Type":"ContainerStarted","Data":"0cb13038ce245f08039edc691504653db141ca0b49451ef599c2b489cfcafa9a"}
Oct 30 13:23:22.190105 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:22.190058    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9e946-8f46bbc55-xmk9s" event={"ID":"8662720a-f907-42b3-a58e-d37b854a9384","Type":"ContainerStarted","Data":"9a1de1abf9d85d930cfd9f61e6c9c6470166f0e8af0b7c57563f432acb820934"}
Oct 30 13:23:22.190539 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:22.190147    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9e946-8f46bbc55-xmk9s"
Oct 30 13:23:22.234470 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:22.234417    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9e946-8f46bbc55-xmk9s" podStartSLOduration=1.9282934809999999 podStartE2EDuration="2.234401516s" totalImagesPullingTime="306.108035ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:23:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:23:22.230506521 +0000 UTC m=+1221.377773762" watchObservedRunningTime="2026-10-30 13:23:22.234401516 +0000 UTC m=+1221.381668757"
Oct 30 13:23:28.203847 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:28.203814    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9e946-8f46bbc55-xmk9s"
Oct 30 13:23:43.954370 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:43.954314    5093 patch_prober.go:29] interesting pod/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:23:43.954962 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:43.954398    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw" podUID="20970fe5-df45-4db7-8d63-6c224f8bf7bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:23:46.784645 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:46.784606    5093 patch_prober.go:29] interesting pod/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 13:23:46.785195 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:46.784680    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95" podUID="ca419c67-141b-4a04-8ee6-95301302c378" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 13:23:47.691454 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:47.691415    5093 generic.go:356] "Generic (PLEG): container finished" podID="ca419c67-141b-4a04-8ee6-95301302c378" containerID="5b70a8c379098929a76f02957700146aa58a065c0ca6ce32a4f4e7d72c53427e" exitCode=0
Oct 30 13:23:47.691641 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:47.691489    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95" event={"ID":"ca419c67-141b-4a04-8ee6-95301302c378","Type":"ContainerDied","Data":"5b70a8c379098929a76f02957700146aa58a065c0ca6ce32a4f4e7d72c53427e"}
Oct 30 13:23:47.692101 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:47.692078    5093 scope.go:122] "RemoveContainer" containerID="5b70a8c379098929a76f02957700146aa58a065c0ca6ce32a4f4e7d72c53427e"
Oct 30 13:23:47.693472 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:47.693427    5093 generic.go:356] "Generic (PLEG): container finished" podID="20970fe5-df45-4db7-8d63-6c224f8bf7bb" containerID="c1c6ab2defcaedba8959f85583421e3c8c6de7d18786ba04c9a02f54c7de30ed" exitCode=0
Oct 30 13:23:47.693545 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:47.693506    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw" event={"ID":"20970fe5-df45-4db7-8d63-6c224f8bf7bb","Type":"ContainerDied","Data":"c1c6ab2defcaedba8959f85583421e3c8c6de7d18786ba04c9a02f54c7de30ed"}
Oct 30 13:23:47.693929 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:47.693910    5093 scope.go:122] "RemoveContainer" containerID="c1c6ab2defcaedba8959f85583421e3c8c6de7d18786ba04c9a02f54c7de30ed"
Oct 30 13:23:47.693929 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:47.693912    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:23:47.694078 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:47.693984    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:23:47.694078 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:47.694018    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:23:47.695530 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:47.695500    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:23:47.695624 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:47.695573    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:23:47.695624 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:47.695607    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:23:48.720158 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:48.717216    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95" event={"ID":"ca419c67-141b-4a04-8ee6-95301302c378","Type":"ContainerStarted","Data":"d352e3cc94776beea1f3b487699b559518c2f961a9e689a652349c138081e793"}
Oct 30 13:23:48.737155 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:48.723268    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw" event={"ID":"20970fe5-df45-4db7-8d63-6c224f8bf7bb","Type":"ContainerStarted","Data":"b9fd5b49fa6ce35745ac41855ff40d1e608479f3d6bb5e1280dbc9b0eb3e90b4"}
Oct 30 13:23:49.221029 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:49.220993    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw"
Oct 30 13:23:49.221211 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:49.221094    5093 patch_prober.go:29] interesting pod/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 13:23:49.221211 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:49.221155    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw" podUID="20970fe5-df45-4db7-8d63-6c224f8bf7bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 13:23:49.647229 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:49.646535    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95"
Oct 30 13:23:49.647229 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:49.646670    5093 patch_prober.go:29] interesting pod/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 13:23:49.647229 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:49.646718    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95" podUID="ca419c67-141b-4a04-8ee6-95301302c378" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 13:23:50.225072 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:50.222790    5093 patch_prober.go:29] interesting pod/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 13:23:50.225072 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:50.222859    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw" podUID="20970fe5-df45-4db7-8d63-6c224f8bf7bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 13:23:50.653740 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:50.653630    5093 patch_prober.go:29] interesting pod/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 13:23:50.654077 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:50.654038    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95" podUID="ca419c67-141b-4a04-8ee6-95301302c378" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 13:23:51.220512 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:51.220359    5093 patch_prober.go:29] interesting pod/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 13:23:51.220512 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:51.220425    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw" podUID="20970fe5-df45-4db7-8d63-6c224f8bf7bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 13:23:51.306418 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:23:51.306368    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:23:51.711225 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:51.645891    5093 patch_prober.go:29] interesting pod/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 13:23:51.711225 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:51.645952    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95" podUID="ca419c67-141b-4a04-8ee6-95301302c378" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 13:23:52.222738 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:52.222685    5093 patch_prober.go:29] interesting pod/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 13:23:52.222989 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:52.222767    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw" podUID="20970fe5-df45-4db7-8d63-6c224f8bf7bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 13:23:52.648758 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:52.648637    5093 patch_prober.go:29] interesting pod/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 13:23:52.649369 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:52.649332    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95" podUID="ca419c67-141b-4a04-8ee6-95301302c378" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 13:23:53.221046 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:53.221012    5093 patch_prober.go:29] interesting pod/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 13:23:53.221240 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:53.221069    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw" podUID="20970fe5-df45-4db7-8d63-6c224f8bf7bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 13:23:53.646568 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:53.646471    5093 patch_prober.go:29] interesting pod/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 13:23:53.649331 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:53.649292    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95" podUID="ca419c67-141b-4a04-8ee6-95301302c378" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 13:23:54.224033 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:54.223566    5093 patch_prober.go:29] interesting pod/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 13:23:54.224033 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:54.223636    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw" podUID="20970fe5-df45-4db7-8d63-6c224f8bf7bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 13:23:54.321107 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:54.321062    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4"]
Oct 30 13:23:54.422559 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:54.422518    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4" podUID="6e9f2890-8091-4bd7-8471-9c7d97081bd3" containerName="istio-proxy" containerID="cri-o://adcda4545958767c492f93d5e4ba68df2e433b22b678cf156199a9ae57577d57" gracePeriod=30
Oct 30 13:23:54.606394 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:54.606285    5093 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv" podUID="d6a44981-104f-4d3e-9937-d184a6ddc482" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:23:54.619236 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:54.616261    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw" podUID="ad37954e-1350-4f75-a63f-a0c0f29ae416" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:23:54.648897 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:54.616318    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg" podUID="ea079ac1-f502-4b5f-b16a-c91e2de98de8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:23:54.651267 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:54.651231    5093 patch_prober.go:29] interesting pod/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 13:23:54.651827 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:54.651794    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95" podUID="ca419c67-141b-4a04-8ee6-95301302c378" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 13:23:55.447553 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:55.447519    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw"
Oct 30 13:23:55.449402 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:55.449377    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw"
Oct 30 13:23:55.480153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:55.479338    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw"
Oct 30 13:23:55.662016 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:55.661964    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95"
Oct 30 13:23:55.663032 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:55.663003    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95"
Oct 30 13:23:55.664810 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:55.664791    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95"
Oct 30 13:23:58.466505 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:58.466459    5093 patch_prober.go:29] interesting pod/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:23:58.467704 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:58.467669    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4" podUID="6e9f2890-8091-4bd7-8471-9c7d97081bd3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:23:59.408185 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:59.408147    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls"]
Oct 30 13:23:59.408431 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:59.408386    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls" podUID="c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a" containerName="istio-proxy" containerID="cri-o://70fd6e41d90aac4da67c626477432abffab291b4d1c96fa15e5a27cec60480d5" gracePeriod=30
Oct 30 13:23:59.422455 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:59.422392    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms"]
Oct 30 13:23:59.422764 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:59.422692    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms" podUID="616525f4-fcee-4d8e-977e-f2104a46208f" containerName="istio-proxy" containerID="cri-o://3048453c7208161b8ae59ceef846024fe85d7e6291ed581c66ed25e55a8efc88" gracePeriod=30
Oct 30 13:23:59.436649 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:59.436198    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9"]
Oct 30 13:23:59.436649 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:59.436449    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9" podUID="1b9ce40f-35d8-4041-a510-37f2cd68d2b3" containerName="istio-proxy" containerID="cri-o://e149248c2ec9bee6cc4de580d5fada481c55f798eca11f4b6e88948a8955c455" gracePeriod=30
Oct 30 13:23:59.464154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:59.458547    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc"]
Oct 30 13:23:59.464154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:59.458798    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc" podUID="8daf18c1-014e-4c6b-bffb-bb25ceef69f3" containerName="istio-proxy" containerID="cri-o://9598582f32e50cedf06d82de1ba545385878ca286d586f3a05e582bedab4db9f" gracePeriod=30
Oct 30 13:23:59.856395 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:59.856350    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4"
Oct 30 13:23:59.984883 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:59.984840    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e9f2890-8091-4bd7-8471-9c7d97081bd3-istio-token\" (UniqueName: \"kubernetes.io/projected/6e9f2890-8091-4bd7-8471-9c7d97081bd3-istio-token\") pod \"6e9f2890-8091-4bd7-8471-9c7d97081bd3\" (UID: \"6e9f2890-8091-4bd7-8471-9c7d97081bd3\") "
Oct 30 13:23:59.985285 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:59.985250    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e9f2890-8091-4bd7-8471-9c7d97081bd3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e9f2890-8091-4bd7-8471-9c7d97081bd3-credential-socket\") pod \"6e9f2890-8091-4bd7-8471-9c7d97081bd3\" (UID: \"6e9f2890-8091-4bd7-8471-9c7d97081bd3\") "
Oct 30 13:23:59.985453 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:59.985429    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e9f2890-8091-4bd7-8471-9c7d97081bd3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e9f2890-8091-4bd7-8471-9c7d97081bd3-istio-envoy\") pod \"6e9f2890-8091-4bd7-8471-9c7d97081bd3\" (UID: \"6e9f2890-8091-4bd7-8471-9c7d97081bd3\") "
Oct 30 13:23:59.985525 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:59.985483    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e9f2890-8091-4bd7-8471-9c7d97081bd3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e9f2890-8091-4bd7-8471-9c7d97081bd3-istiod-ca-cert\") pod \"6e9f2890-8091-4bd7-8471-9c7d97081bd3\" (UID: \"6e9f2890-8091-4bd7-8471-9c7d97081bd3\") "
Oct 30 13:23:59.985584 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:59.985521    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e9f2890-8091-4bd7-8471-9c7d97081bd3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e9f2890-8091-4bd7-8471-9c7d97081bd3-workload-certs\") pod \"6e9f2890-8091-4bd7-8471-9c7d97081bd3\" (UID: \"6e9f2890-8091-4bd7-8471-9c7d97081bd3\") "
Oct 30 13:23:59.985638 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:59.985604    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e9f2890-8091-4bd7-8471-9c7d97081bd3-kube-api-access-nnlqf\" (UniqueName: \"kubernetes.io/projected/6e9f2890-8091-4bd7-8471-9c7d97081bd3-kube-api-access-nnlqf\") pod \"6e9f2890-8091-4bd7-8471-9c7d97081bd3\" (UID: \"6e9f2890-8091-4bd7-8471-9c7d97081bd3\") "
Oct 30 13:23:59.985712 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:59.985673    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e9f2890-8091-4bd7-8471-9c7d97081bd3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e9f2890-8091-4bd7-8471-9c7d97081bd3-istio-data\") pod \"6e9f2890-8091-4bd7-8471-9c7d97081bd3\" (UID: \"6e9f2890-8091-4bd7-8471-9c7d97081bd3\") "
Oct 30 13:23:59.985775 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:59.985753    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6e9f2890-8091-4bd7-8471-9c7d97081bd3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e9f2890-8091-4bd7-8471-9c7d97081bd3-istio-podinfo\") pod \"6e9f2890-8091-4bd7-8471-9c7d97081bd3\" (UID: \"6e9f2890-8091-4bd7-8471-9c7d97081bd3\") "
Oct 30 13:23:59.985824 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:59.985785    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e9f2890-8091-4bd7-8471-9c7d97081bd3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e9f2890-8091-4bd7-8471-9c7d97081bd3-workload-socket\") pod \"6e9f2890-8091-4bd7-8471-9c7d97081bd3\" (UID: \"6e9f2890-8091-4bd7-8471-9c7d97081bd3\") "
Oct 30 13:23:59.986933 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:59.985978    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e9f2890-8091-4bd7-8471-9c7d97081bd3-credential-socket" pod "6e9f2890-8091-4bd7-8471-9c7d97081bd3" (UID: "6e9f2890-8091-4bd7-8471-9c7d97081bd3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:23:59.986933 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:59.986321    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e9f2890-8091-4bd7-8471-9c7d97081bd3-istio-data" pod "6e9f2890-8091-4bd7-8471-9c7d97081bd3" (UID: "6e9f2890-8091-4bd7-8471-9c7d97081bd3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:23:59.986933 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:59.986336    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e9f2890-8091-4bd7-8471-9c7d97081bd3-workload-certs" pod "6e9f2890-8091-4bd7-8471-9c7d97081bd3" (UID: "6e9f2890-8091-4bd7-8471-9c7d97081bd3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:23:59.986933 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:59.986625    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e9f2890-8091-4bd7-8471-9c7d97081bd3-istiod-ca-cert" pod "6e9f2890-8091-4bd7-8471-9c7d97081bd3" (UID: "6e9f2890-8091-4bd7-8471-9c7d97081bd3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:23:59.986933 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:59.986786    5093 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e9f2890-8091-4bd7-8471-9c7d97081bd3-credential-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:23:59.986933 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:59.986800    5093 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e9f2890-8091-4bd7-8471-9c7d97081bd3-istiod-ca-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:23:59.986933 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:59.986811    5093 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e9f2890-8091-4bd7-8471-9c7d97081bd3-workload-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:23:59.986933 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:59.986824    5093 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e9f2890-8091-4bd7-8471-9c7d97081bd3-istio-data\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:23:59.986933 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:59.986883    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e9f2890-8091-4bd7-8471-9c7d97081bd3-workload-socket" pod "6e9f2890-8091-4bd7-8471-9c7d97081bd3" (UID: "6e9f2890-8091-4bd7-8471-9c7d97081bd3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:23:59.988380 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:59.988346    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e9f2890-8091-4bd7-8471-9c7d97081bd3-istio-token" pod "6e9f2890-8091-4bd7-8471-9c7d97081bd3" (UID: "6e9f2890-8091-4bd7-8471-9c7d97081bd3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:23:59.988565 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:59.988504    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e9f2890-8091-4bd7-8471-9c7d97081bd3-istio-podinfo" pod "6e9f2890-8091-4bd7-8471-9c7d97081bd3" (UID: "6e9f2890-8091-4bd7-8471-9c7d97081bd3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:23:59.988565 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:59.988537    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e9f2890-8091-4bd7-8471-9c7d97081bd3-istio-envoy" pod "6e9f2890-8091-4bd7-8471-9c7d97081bd3" (UID: "6e9f2890-8091-4bd7-8471-9c7d97081bd3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:23:59.988806 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:23:59.988776    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e9f2890-8091-4bd7-8471-9c7d97081bd3-kube-api-access-nnlqf" pod "6e9f2890-8091-4bd7-8471-9c7d97081bd3" (UID: "6e9f2890-8091-4bd7-8471-9c7d97081bd3"). InnerVolumeSpecName "kube-api-access-nnlqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:00.098076 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:00.095044    5093 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e9f2890-8091-4bd7-8471-9c7d97081bd3-istio-podinfo\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:00.098076 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:00.095084    5093 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e9f2890-8091-4bd7-8471-9c7d97081bd3-workload-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:00.098076 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:00.095107    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e9f2890-8091-4bd7-8471-9c7d97081bd3-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:00.099157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:00.098467    5093 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e9f2890-8091-4bd7-8471-9c7d97081bd3-istio-envoy\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:00.099157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:00.098499    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnlqf\" (UniqueName: \"kubernetes.io/projected/6e9f2890-8091-4bd7-8471-9c7d97081bd3-kube-api-access-nnlqf\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:00.120083 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:00.120040    5093 generic.go:356] "Generic (PLEG): container finished" podID="6e9f2890-8091-4bd7-8471-9c7d97081bd3" containerID="adcda4545958767c492f93d5e4ba68df2e433b22b678cf156199a9ae57577d57" exitCode=0
Oct 30 13:24:00.162370 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:00.120101    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4"
Oct 30 13:24:00.165642 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:00.120120    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4" event={"ID":"6e9f2890-8091-4bd7-8471-9c7d97081bd3","Type":"ContainerDied","Data":"adcda4545958767c492f93d5e4ba68df2e433b22b678cf156199a9ae57577d57"}
Oct 30 13:24:00.165922 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:00.165897    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4" event={"ID":"6e9f2890-8091-4bd7-8471-9c7d97081bd3","Type":"ContainerDied","Data":"2a45db66db571f860408fbb15cb9b24ff16cf112651f31f0826cd870ff00bdf9"}
Oct 30 13:24:00.166060 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:00.166046    5093 scope.go:122] "RemoveContainer" containerID="adcda4545958767c492f93d5e4ba68df2e433b22b678cf156199a9ae57577d57"
Oct 30 13:24:00.189558 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:00.189532    5093 scope.go:122] "RemoveContainer" containerID="47fc90b4cc96de627928b8490fac0061319075dafcb5d66cd2d7eb5ee2177df1"
Oct 30 13:24:00.201867 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:00.201769    5093 scope.go:122] "RemoveContainer" containerID="adcda4545958767c492f93d5e4ba68df2e433b22b678cf156199a9ae57577d57"
Oct 30 13:24:00.202190 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:24:00.202147    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adcda4545958767c492f93d5e4ba68df2e433b22b678cf156199a9ae57577d57\": container with ID starting with adcda4545958767c492f93d5e4ba68df2e433b22b678cf156199a9ae57577d57 not found: ID does not exist" containerID="adcda4545958767c492f93d5e4ba68df2e433b22b678cf156199a9ae57577d57"
Oct 30 13:24:00.202304 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:00.202190    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adcda4545958767c492f93d5e4ba68df2e433b22b678cf156199a9ae57577d57"} err="failed to get container status \"adcda4545958767c492f93d5e4ba68df2e433b22b678cf156199a9ae57577d57\": rpc error: code = NotFound desc = could not find container \"adcda4545958767c492f93d5e4ba68df2e433b22b678cf156199a9ae57577d57\": container with ID starting with adcda4545958767c492f93d5e4ba68df2e433b22b678cf156199a9ae57577d57 not found: ID does not exist"
Oct 30 13:24:00.202304 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:00.202219    5093 scope.go:122] "RemoveContainer" containerID="47fc90b4cc96de627928b8490fac0061319075dafcb5d66cd2d7eb5ee2177df1"
Oct 30 13:24:00.202498 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:24:00.202478    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47fc90b4cc96de627928b8490fac0061319075dafcb5d66cd2d7eb5ee2177df1\": container with ID starting with 47fc90b4cc96de627928b8490fac0061319075dafcb5d66cd2d7eb5ee2177df1 not found: ID does not exist" containerID="47fc90b4cc96de627928b8490fac0061319075dafcb5d66cd2d7eb5ee2177df1"
Oct 30 13:24:00.202561 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:00.202523    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47fc90b4cc96de627928b8490fac0061319075dafcb5d66cd2d7eb5ee2177df1"} err="failed to get container status \"47fc90b4cc96de627928b8490fac0061319075dafcb5d66cd2d7eb5ee2177df1\": rpc error: code = NotFound desc = could not find container \"47fc90b4cc96de627928b8490fac0061319075dafcb5d66cd2d7eb5ee2177df1\": container with ID starting with 47fc90b4cc96de627928b8490fac0061319075dafcb5d66cd2d7eb5ee2177df1 not found: ID does not exist"
Oct 30 13:24:00.284162 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:00.276582    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4"]
Oct 30 13:24:00.284162 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:00.280924    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-7c0e28-openshift-default-6fddc9d5fc-sv9z4"]
Oct 30 13:24:01.174431 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:01.173778    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e9f2890-8091-4bd7-8471-9c7d97081bd3" path="/var/lib/kubelet/pods/6e9f2890-8091-4bd7-8471-9c7d97081bd3/volumes"
Oct 30 13:24:04.018195 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:04.014349    5093 patch_prober.go:29] interesting pod/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:24:04.018195 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:04.014437    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf" podUID="58244ace-ebd4-4b98-929b-8396655992a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:24:04.782054 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:04.782023    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms"
Oct 30 13:24:04.823401 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:04.823341    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9"
Oct 30 13:24:04.901649 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:04.890822    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/616525f4-fcee-4d8e-977e-f2104a46208f-istio-token\" (UniqueName: \"kubernetes.io/projected/616525f4-fcee-4d8e-977e-f2104a46208f-istio-token\") pod \"616525f4-fcee-4d8e-977e-f2104a46208f\" (UID: \"616525f4-fcee-4d8e-977e-f2104a46208f\") "
Oct 30 13:24:04.901649 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:04.890882    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/616525f4-fcee-4d8e-977e-f2104a46208f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/616525f4-fcee-4d8e-977e-f2104a46208f-credential-socket\") pod \"616525f4-fcee-4d8e-977e-f2104a46208f\" (UID: \"616525f4-fcee-4d8e-977e-f2104a46208f\") "
Oct 30 13:24:04.901649 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:04.890926    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/616525f4-fcee-4d8e-977e-f2104a46208f-kube-api-access-rjkxl\" (UniqueName: \"kubernetes.io/projected/616525f4-fcee-4d8e-977e-f2104a46208f-kube-api-access-rjkxl\") pod \"616525f4-fcee-4d8e-977e-f2104a46208f\" (UID: \"616525f4-fcee-4d8e-977e-f2104a46208f\") "
Oct 30 13:24:04.901649 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:04.890996    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/616525f4-fcee-4d8e-977e-f2104a46208f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/616525f4-fcee-4d8e-977e-f2104a46208f-istio-podinfo\") pod \"616525f4-fcee-4d8e-977e-f2104a46208f\" (UID: \"616525f4-fcee-4d8e-977e-f2104a46208f\") "
Oct 30 13:24:04.901649 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:04.891051    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/616525f4-fcee-4d8e-977e-f2104a46208f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/616525f4-fcee-4d8e-977e-f2104a46208f-istio-data\") pod \"616525f4-fcee-4d8e-977e-f2104a46208f\" (UID: \"616525f4-fcee-4d8e-977e-f2104a46208f\") "
Oct 30 13:24:04.901649 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:04.891115    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/616525f4-fcee-4d8e-977e-f2104a46208f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/616525f4-fcee-4d8e-977e-f2104a46208f-istiod-ca-cert\") pod \"616525f4-fcee-4d8e-977e-f2104a46208f\" (UID: \"616525f4-fcee-4d8e-977e-f2104a46208f\") "
Oct 30 13:24:04.901649 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:04.891156    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/616525f4-fcee-4d8e-977e-f2104a46208f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/616525f4-fcee-4d8e-977e-f2104a46208f-workload-socket\") pod \"616525f4-fcee-4d8e-977e-f2104a46208f\" (UID: \"616525f4-fcee-4d8e-977e-f2104a46208f\") "
Oct 30 13:24:04.901649 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:04.891225    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/616525f4-fcee-4d8e-977e-f2104a46208f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/616525f4-fcee-4d8e-977e-f2104a46208f-workload-certs\") pod \"616525f4-fcee-4d8e-977e-f2104a46208f\" (UID: \"616525f4-fcee-4d8e-977e-f2104a46208f\") "
Oct 30 13:24:04.901649 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:04.891345    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/616525f4-fcee-4d8e-977e-f2104a46208f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/616525f4-fcee-4d8e-977e-f2104a46208f-istio-envoy\") pod \"616525f4-fcee-4d8e-977e-f2104a46208f\" (UID: \"616525f4-fcee-4d8e-977e-f2104a46208f\") "
Oct 30 13:24:04.901649 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:04.893857    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616525f4-fcee-4d8e-977e-f2104a46208f-workload-socket" pod "616525f4-fcee-4d8e-977e-f2104a46208f" (UID: "616525f4-fcee-4d8e-977e-f2104a46208f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:04.901649 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:04.894453    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616525f4-fcee-4d8e-977e-f2104a46208f-workload-certs" pod "616525f4-fcee-4d8e-977e-f2104a46208f" (UID: "616525f4-fcee-4d8e-977e-f2104a46208f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:04.901649 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:04.894519    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616525f4-fcee-4d8e-977e-f2104a46208f-istio-data" pod "616525f4-fcee-4d8e-977e-f2104a46208f" (UID: "616525f4-fcee-4d8e-977e-f2104a46208f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:04.901649 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:04.894574    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616525f4-fcee-4d8e-977e-f2104a46208f-credential-socket" pod "616525f4-fcee-4d8e-977e-f2104a46208f" (UID: "616525f4-fcee-4d8e-977e-f2104a46208f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:04.901649 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:04.894856    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/616525f4-fcee-4d8e-977e-f2104a46208f-istiod-ca-cert" pod "616525f4-fcee-4d8e-977e-f2104a46208f" (UID: "616525f4-fcee-4d8e-977e-f2104a46208f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:24:04.901649 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:04.897920    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/616525f4-fcee-4d8e-977e-f2104a46208f-istio-podinfo" pod "616525f4-fcee-4d8e-977e-f2104a46208f" (UID: "616525f4-fcee-4d8e-977e-f2104a46208f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:24:04.901649 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:04.898275    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/616525f4-fcee-4d8e-977e-f2104a46208f-istio-token" pod "616525f4-fcee-4d8e-977e-f2104a46208f" (UID: "616525f4-fcee-4d8e-977e-f2104a46208f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:04.901649 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:04.900056    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/616525f4-fcee-4d8e-977e-f2104a46208f-kube-api-access-rjkxl" pod "616525f4-fcee-4d8e-977e-f2104a46208f" (UID: "616525f4-fcee-4d8e-977e-f2104a46208f"). InnerVolumeSpecName "kube-api-access-rjkxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:04.901649 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:04.900081    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616525f4-fcee-4d8e-977e-f2104a46208f-istio-envoy" pod "616525f4-fcee-4d8e-977e-f2104a46208f" (UID: "616525f4-fcee-4d8e-977e-f2104a46208f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:04.963585 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:04.963556    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc"
Oct 30 13:24:04.967779 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:04.967710    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls"
Oct 30 13:24:05.006774 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:04.993854    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-istiod-ca-cert\") pod \"1b9ce40f-35d8-4041-a510-37f2cd68d2b3\" (UID: \"1b9ce40f-35d8-4041-a510-37f2cd68d2b3\") "
Oct 30 13:24:05.006774 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:04.993965    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-workload-socket\") pod \"1b9ce40f-35d8-4041-a510-37f2cd68d2b3\" (UID: \"1b9ce40f-35d8-4041-a510-37f2cd68d2b3\") "
Oct 30 13:24:05.006774 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:04.994044    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-istio-token\" (UniqueName: \"kubernetes.io/projected/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-istio-token\") pod \"1b9ce40f-35d8-4041-a510-37f2cd68d2b3\" (UID: \"1b9ce40f-35d8-4041-a510-37f2cd68d2b3\") "
Oct 30 13:24:05.006774 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:04.994087    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-credential-socket\") pod \"1b9ce40f-35d8-4041-a510-37f2cd68d2b3\" (UID: \"1b9ce40f-35d8-4041-a510-37f2cd68d2b3\") "
Oct 30 13:24:05.006774 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:04.994123    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-istio-data\") pod \"1b9ce40f-35d8-4041-a510-37f2cd68d2b3\" (UID: \"1b9ce40f-35d8-4041-a510-37f2cd68d2b3\") "
Oct 30 13:24:05.006774 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:04.994210    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-kube-api-access-q6k2c\" (UniqueName: \"kubernetes.io/projected/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-kube-api-access-q6k2c\") pod \"1b9ce40f-35d8-4041-a510-37f2cd68d2b3\" (UID: \"1b9ce40f-35d8-4041-a510-37f2cd68d2b3\") "
Oct 30 13:24:05.006774 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:04.994467    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-istiod-ca-cert" pod "1b9ce40f-35d8-4041-a510-37f2cd68d2b3" (UID: "1b9ce40f-35d8-4041-a510-37f2cd68d2b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:24:05.006774 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:04.994578    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-istio-podinfo\") pod \"1b9ce40f-35d8-4041-a510-37f2cd68d2b3\" (UID: \"1b9ce40f-35d8-4041-a510-37f2cd68d2b3\") "
Oct 30 13:24:05.006774 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:04.994674    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-workload-certs\") pod \"1b9ce40f-35d8-4041-a510-37f2cd68d2b3\" (UID: \"1b9ce40f-35d8-4041-a510-37f2cd68d2b3\") "
Oct 30 13:24:05.006774 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:04.994757    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-istio-envoy\") pod \"1b9ce40f-35d8-4041-a510-37f2cd68d2b3\" (UID: \"1b9ce40f-35d8-4041-a510-37f2cd68d2b3\") "
Oct 30 13:24:05.006774 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:04.995644    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/616525f4-fcee-4d8e-977e-f2104a46208f-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:05.006774 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:04.995667    5093 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/616525f4-fcee-4d8e-977e-f2104a46208f-credential-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:05.006774 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:04.995683    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjkxl\" (UniqueName: \"kubernetes.io/projected/616525f4-fcee-4d8e-977e-f2104a46208f-kube-api-access-rjkxl\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:05.006774 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:04.995698    5093 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/616525f4-fcee-4d8e-977e-f2104a46208f-istio-podinfo\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:05.006774 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:04.995711    5093 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-istiod-ca-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:05.006774 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:04.995722    5093 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/616525f4-fcee-4d8e-977e-f2104a46208f-istio-data\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:05.006774 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:04.995733    5093 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/616525f4-fcee-4d8e-977e-f2104a46208f-istiod-ca-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:05.006774 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:04.995746    5093 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/616525f4-fcee-4d8e-977e-f2104a46208f-workload-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:05.006774 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:04.995760    5093 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/616525f4-fcee-4d8e-977e-f2104a46208f-workload-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:05.006774 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:04.995772    5093 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/616525f4-fcee-4d8e-977e-f2104a46208f-istio-envoy\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:05.006774 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:04.996447    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-workload-socket" pod "1b9ce40f-35d8-4041-a510-37f2cd68d2b3" (UID: "1b9ce40f-35d8-4041-a510-37f2cd68d2b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:05.006774 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:04.997585    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-credential-socket" pod "1b9ce40f-35d8-4041-a510-37f2cd68d2b3" (UID: "1b9ce40f-35d8-4041-a510-37f2cd68d2b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:05.006774 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:04.998043    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-istio-data" pod "1b9ce40f-35d8-4041-a510-37f2cd68d2b3" (UID: "1b9ce40f-35d8-4041-a510-37f2cd68d2b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:05.006774 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:04.998592    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-workload-certs" pod "1b9ce40f-35d8-4041-a510-37f2cd68d2b3" (UID: "1b9ce40f-35d8-4041-a510-37f2cd68d2b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:05.006774 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.003111    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-kube-api-access-q6k2c" pod "1b9ce40f-35d8-4041-a510-37f2cd68d2b3" (UID: "1b9ce40f-35d8-4041-a510-37f2cd68d2b3"). InnerVolumeSpecName "kube-api-access-q6k2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:05.006774 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.004260    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-istio-envoy" pod "1b9ce40f-35d8-4041-a510-37f2cd68d2b3" (UID: "1b9ce40f-35d8-4041-a510-37f2cd68d2b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:05.006774 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.006749    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-istio-podinfo" pod "1b9ce40f-35d8-4041-a510-37f2cd68d2b3" (UID: "1b9ce40f-35d8-4041-a510-37f2cd68d2b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:24:05.010196 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.009054    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-istio-token" pod "1b9ce40f-35d8-4041-a510-37f2cd68d2b3" (UID: "1b9ce40f-35d8-4041-a510-37f2cd68d2b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:05.096975 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.096936    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-workload-socket\") pod \"c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a\" (UID: \"c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a\") "
Oct 30 13:24:05.097509 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.097014    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-kube-api-access-5jf5k\" (UniqueName: \"kubernetes.io/projected/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-kube-api-access-5jf5k\") pod \"c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a\" (UID: \"c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a\") "
Oct 30 13:24:05.097509 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.097048    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-istio-token\" (UniqueName: \"kubernetes.io/projected/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-istio-token\") pod \"c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a\" (UID: \"c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a\") "
Oct 30 13:24:05.097509 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.097111    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-istiod-ca-cert\") pod \"c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a\" (UID: \"c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a\") "
Oct 30 13:24:05.097509 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.097227    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-istio-envoy\") pod \"8daf18c1-014e-4c6b-bffb-bb25ceef69f3\" (UID: \"8daf18c1-014e-4c6b-bffb-bb25ceef69f3\") "
Oct 30 13:24:05.097509 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.097285    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-credential-socket\") pod \"8daf18c1-014e-4c6b-bffb-bb25ceef69f3\" (UID: \"8daf18c1-014e-4c6b-bffb-bb25ceef69f3\") "
Oct 30 13:24:05.097509 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.097318    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-workload-socket" pod "c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a" (UID: "c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:05.097509 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.097381    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-workload-certs\") pod \"8daf18c1-014e-4c6b-bffb-bb25ceef69f3\" (UID: \"8daf18c1-014e-4c6b-bffb-bb25ceef69f3\") "
Oct 30 13:24:05.097509 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.097423    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-workload-socket\") pod \"8daf18c1-014e-4c6b-bffb-bb25ceef69f3\" (UID: \"8daf18c1-014e-4c6b-bffb-bb25ceef69f3\") "
Oct 30 13:24:05.097509 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.097456    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-credential-socket\") pod \"c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a\" (UID: \"c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a\") "
Oct 30 13:24:05.097509 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.097491    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-istio-envoy\") pod \"c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a\" (UID: \"c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a\") "
Oct 30 13:24:05.098041 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.097560    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-kube-api-access-vvk4j\" (UniqueName: \"kubernetes.io/projected/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-kube-api-access-vvk4j\") pod \"8daf18c1-014e-4c6b-bffb-bb25ceef69f3\" (UID: \"8daf18c1-014e-4c6b-bffb-bb25ceef69f3\") "
Oct 30 13:24:05.098041 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.097641    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-istio-podinfo\") pod \"8daf18c1-014e-4c6b-bffb-bb25ceef69f3\" (UID: \"8daf18c1-014e-4c6b-bffb-bb25ceef69f3\") "
Oct 30 13:24:05.098041 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.097704    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-workload-certs\") pod \"c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a\" (UID: \"c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a\") "
Oct 30 13:24:05.098041 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.097773    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-istio-data\") pod \"c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a\" (UID: \"c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a\") "
Oct 30 13:24:05.098041 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.097828    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-istiod-ca-cert\") pod \"8daf18c1-014e-4c6b-bffb-bb25ceef69f3\" (UID: \"8daf18c1-014e-4c6b-bffb-bb25ceef69f3\") "
Oct 30 13:24:05.098041 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.097884    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-istio-data\") pod \"8daf18c1-014e-4c6b-bffb-bb25ceef69f3\" (UID: \"8daf18c1-014e-4c6b-bffb-bb25ceef69f3\") "
Oct 30 13:24:05.098041 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.097924    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-istio-token\" (UniqueName: \"kubernetes.io/projected/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-istio-token\") pod \"8daf18c1-014e-4c6b-bffb-bb25ceef69f3\" (UID: \"8daf18c1-014e-4c6b-bffb-bb25ceef69f3\") "
Oct 30 13:24:05.098041 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.097950    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-istio-podinfo\") pod \"c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a\" (UID: \"c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a\") "
Oct 30 13:24:05.099065 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.099016    5093 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-istio-envoy\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:05.099065 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.099047    5093 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-workload-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:05.099065 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.099062    5093 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-workload-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:05.099465 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.099076    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:05.099465 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.099090    5093 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-credential-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:05.099465 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.099103    5093 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-istio-data\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:05.099465 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.099116    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6k2c\" (UniqueName: \"kubernetes.io/projected/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-kube-api-access-q6k2c\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:05.099465 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.099146    5093 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-istio-podinfo\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:05.099465 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.099161    5093 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b9ce40f-35d8-4041-a510-37f2cd68d2b3-workload-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:05.100172 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.100141    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-istio-token" pod "c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a" (UID: "c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:05.100332 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.100238    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-istiod-ca-cert" pod "c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a" (UID: "c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:24:05.101551 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.101523    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-istio-podinfo" pod "c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a" (UID: "c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:24:05.101912 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.101888    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-istiod-ca-cert" pod "8daf18c1-014e-4c6b-bffb-bb25ceef69f3" (UID: "8daf18c1-014e-4c6b-bffb-bb25ceef69f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:24:05.102186 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.102165    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-workload-certs" pod "c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a" (UID: "c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:05.102484 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.102401    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-istio-data" pod "c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a" (UID: "c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:05.102645 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.102624    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-workload-socket" pod "8daf18c1-014e-4c6b-bffb-bb25ceef69f3" (UID: "8daf18c1-014e-4c6b-bffb-bb25ceef69f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:05.102827 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.102806    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-credential-socket" pod "8daf18c1-014e-4c6b-bffb-bb25ceef69f3" (UID: "8daf18c1-014e-4c6b-bffb-bb25ceef69f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:05.103038 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.103014    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-workload-certs" pod "8daf18c1-014e-4c6b-bffb-bb25ceef69f3" (UID: "8daf18c1-014e-4c6b-bffb-bb25ceef69f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:05.103276 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.103077    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-istio-envoy" pod "8daf18c1-014e-4c6b-bffb-bb25ceef69f3" (UID: "8daf18c1-014e-4c6b-bffb-bb25ceef69f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:05.103371 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.103282    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-istio-data" pod "8daf18c1-014e-4c6b-bffb-bb25ceef69f3" (UID: "8daf18c1-014e-4c6b-bffb-bb25ceef69f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:05.103371 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.103354    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-credential-socket" pod "c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a" (UID: "c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:05.105101 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.105068    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-istio-token" pod "8daf18c1-014e-4c6b-bffb-bb25ceef69f3" (UID: "8daf18c1-014e-4c6b-bffb-bb25ceef69f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:05.105295 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.105270    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-istio-podinfo" pod "8daf18c1-014e-4c6b-bffb-bb25ceef69f3" (UID: "8daf18c1-014e-4c6b-bffb-bb25ceef69f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:24:05.106091 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.106049    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-kube-api-access-vvk4j" pod "8daf18c1-014e-4c6b-bffb-bb25ceef69f3" (UID: "8daf18c1-014e-4c6b-bffb-bb25ceef69f3"). InnerVolumeSpecName "kube-api-access-vvk4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:05.106348 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.106316    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-istio-envoy" pod "c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a" (UID: "c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:05.108001 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.107828    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-kube-api-access-5jf5k" pod "c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a" (UID: "c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a"). InnerVolumeSpecName "kube-api-access-5jf5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:05.200695 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.200645    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jf5k\" (UniqueName: \"kubernetes.io/projected/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-kube-api-access-5jf5k\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:05.200695 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.200679    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:05.200695 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.200689    5093 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-istiod-ca-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:05.200695 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.200698    5093 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-istio-envoy\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:05.200695 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.200706    5093 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-credential-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:05.201117 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.200715    5093 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-workload-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:05.201117 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.200723    5093 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-workload-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:05.201117 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.200731    5093 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-credential-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:05.201117 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.200739    5093 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-istio-envoy\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:05.201117 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.200747    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvk4j\" (UniqueName: \"kubernetes.io/projected/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-kube-api-access-vvk4j\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:05.201117 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.200756    5093 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-istio-podinfo\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:05.201117 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.200764    5093 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-workload-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:05.201117 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.200771    5093 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-istio-data\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:05.201117 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.200779    5093 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-istiod-ca-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:05.201117 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.200798    5093 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-istio-data\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:05.201117 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.200808    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8daf18c1-014e-4c6b-bffb-bb25ceef69f3-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:05.201117 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.200816    5093 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a-istio-podinfo\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:05.232068 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.231971    5093 generic.go:356] "Generic (PLEG): container finished" podID="616525f4-fcee-4d8e-977e-f2104a46208f" containerID="3048453c7208161b8ae59ceef846024fe85d7e6291ed581c66ed25e55a8efc88" exitCode=0
Oct 30 13:24:05.232068 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.232016    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms"
Oct 30 13:24:05.232068 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.232051    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms" event={"ID":"616525f4-fcee-4d8e-977e-f2104a46208f","Type":"ContainerDied","Data":"3048453c7208161b8ae59ceef846024fe85d7e6291ed581c66ed25e55a8efc88"}
Oct 30 13:24:05.232450 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.232105    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms" event={"ID":"616525f4-fcee-4d8e-977e-f2104a46208f","Type":"ContainerDied","Data":"e901c49c268969bfaff3b2358c5239b7c0f1b893d046cf0b333932cac4c6ff7a"}
Oct 30 13:24:05.232450 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.232169    5093 scope.go:122] "RemoveContainer" containerID="3048453c7208161b8ae59ceef846024fe85d7e6291ed581c66ed25e55a8efc88"
Oct 30 13:24:05.237393 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.234030    5093 generic.go:356] "Generic (PLEG): container finished" podID="1b9ce40f-35d8-4041-a510-37f2cd68d2b3" containerID="e149248c2ec9bee6cc4de580d5fada481c55f798eca11f4b6e88948a8955c455" exitCode=0
Oct 30 13:24:05.237393 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.234371    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9"
Oct 30 13:24:05.237393 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.234975    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9" event={"ID":"1b9ce40f-35d8-4041-a510-37f2cd68d2b3","Type":"ContainerDied","Data":"e149248c2ec9bee6cc4de580d5fada481c55f798eca11f4b6e88948a8955c455"}
Oct 30 13:24:05.237393 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.235025    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9" event={"ID":"1b9ce40f-35d8-4041-a510-37f2cd68d2b3","Type":"ContainerDied","Data":"e0b879742234f48543163c3aed7b59b91ab64e9e12390a48b22271d5caeb7ffb"}
Oct 30 13:24:05.237393 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.236982    5093 generic.go:356] "Generic (PLEG): container finished" podID="8daf18c1-014e-4c6b-bffb-bb25ceef69f3" containerID="9598582f32e50cedf06d82de1ba545385878ca286d586f3a05e582bedab4db9f" exitCode=0
Oct 30 13:24:05.237393 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.237030    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc" event={"ID":"8daf18c1-014e-4c6b-bffb-bb25ceef69f3","Type":"ContainerDied","Data":"9598582f32e50cedf06d82de1ba545385878ca286d586f3a05e582bedab4db9f"}
Oct 30 13:24:05.237393 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.237112    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc" event={"ID":"8daf18c1-014e-4c6b-bffb-bb25ceef69f3","Type":"ContainerDied","Data":"2701fd856b5045dd9505a9b6fa015cbd4270105b6869939175eae5a1239fc600"}
Oct 30 13:24:05.237393 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.237041    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc"
Oct 30 13:24:05.244810 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.244761    5093 generic.go:356] "Generic (PLEG): container finished" podID="c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a" containerID="70fd6e41d90aac4da67c626477432abffab291b4d1c96fa15e5a27cec60480d5" exitCode=0
Oct 30 13:24:05.244952 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.244835    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls" event={"ID":"c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a","Type":"ContainerDied","Data":"70fd6e41d90aac4da67c626477432abffab291b4d1c96fa15e5a27cec60480d5"}
Oct 30 13:24:05.244952 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.244883    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls" event={"ID":"c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a","Type":"ContainerDied","Data":"0af2b6abc6ed57a754be68eaaa83de04d14de13bd36b44b30c4ee443102f1772"}
Oct 30 13:24:05.245082 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.244965    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls"
Oct 30 13:24:05.249433 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.248908    5093 scope.go:122] "RemoveContainer" containerID="3048453c7208161b8ae59ceef846024fe85d7e6291ed581c66ed25e55a8efc88"
Oct 30 13:24:05.250978 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:24:05.250821    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3048453c7208161b8ae59ceef846024fe85d7e6291ed581c66ed25e55a8efc88\": container with ID starting with 3048453c7208161b8ae59ceef846024fe85d7e6291ed581c66ed25e55a8efc88 not found: ID does not exist" containerID="3048453c7208161b8ae59ceef846024fe85d7e6291ed581c66ed25e55a8efc88"
Oct 30 13:24:05.250978 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.250863    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3048453c7208161b8ae59ceef846024fe85d7e6291ed581c66ed25e55a8efc88"} err="failed to get container status \"3048453c7208161b8ae59ceef846024fe85d7e6291ed581c66ed25e55a8efc88\": rpc error: code = NotFound desc = could not find container \"3048453c7208161b8ae59ceef846024fe85d7e6291ed581c66ed25e55a8efc88\": container with ID starting with 3048453c7208161b8ae59ceef846024fe85d7e6291ed581c66ed25e55a8efc88 not found: ID does not exist"
Oct 30 13:24:05.250978 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.250886    5093 scope.go:122] "RemoveContainer" containerID="e149248c2ec9bee6cc4de580d5fada481c55f798eca11f4b6e88948a8955c455"
Oct 30 13:24:05.263803 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.263777    5093 scope.go:122] "RemoveContainer" containerID="e149248c2ec9bee6cc4de580d5fada481c55f798eca11f4b6e88948a8955c455"
Oct 30 13:24:05.264103 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:24:05.264083    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e149248c2ec9bee6cc4de580d5fada481c55f798eca11f4b6e88948a8955c455\": container with ID starting with e149248c2ec9bee6cc4de580d5fada481c55f798eca11f4b6e88948a8955c455 not found: ID does not exist" containerID="e149248c2ec9bee6cc4de580d5fada481c55f798eca11f4b6e88948a8955c455"
Oct 30 13:24:05.264327 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.264115    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e149248c2ec9bee6cc4de580d5fada481c55f798eca11f4b6e88948a8955c455"} err="failed to get container status \"e149248c2ec9bee6cc4de580d5fada481c55f798eca11f4b6e88948a8955c455\": rpc error: code = NotFound desc = could not find container \"e149248c2ec9bee6cc4de580d5fada481c55f798eca11f4b6e88948a8955c455\": container with ID starting with e149248c2ec9bee6cc4de580d5fada481c55f798eca11f4b6e88948a8955c455 not found: ID does not exist"
Oct 30 13:24:05.264327 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.264208    5093 scope.go:122] "RemoveContainer" containerID="9598582f32e50cedf06d82de1ba545385878ca286d586f3a05e582bedab4db9f"
Oct 30 13:24:05.277032 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.277007    5093 scope.go:122] "RemoveContainer" containerID="9598582f32e50cedf06d82de1ba545385878ca286d586f3a05e582bedab4db9f"
Oct 30 13:24:05.277456 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:24:05.277423    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9598582f32e50cedf06d82de1ba545385878ca286d586f3a05e582bedab4db9f\": container with ID starting with 9598582f32e50cedf06d82de1ba545385878ca286d586f3a05e582bedab4db9f not found: ID does not exist" containerID="9598582f32e50cedf06d82de1ba545385878ca286d586f3a05e582bedab4db9f"
Oct 30 13:24:05.277554 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.277462    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9598582f32e50cedf06d82de1ba545385878ca286d586f3a05e582bedab4db9f"} err="failed to get container status \"9598582f32e50cedf06d82de1ba545385878ca286d586f3a05e582bedab4db9f\": rpc error: code = NotFound desc = could not find container \"9598582f32e50cedf06d82de1ba545385878ca286d586f3a05e582bedab4db9f\": container with ID starting with 9598582f32e50cedf06d82de1ba545385878ca286d586f3a05e582bedab4db9f not found: ID does not exist"
Oct 30 13:24:05.277554 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.277481    5093 scope.go:122] "RemoveContainer" containerID="70fd6e41d90aac4da67c626477432abffab291b4d1c96fa15e5a27cec60480d5"
Oct 30 13:24:05.290327 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.290290    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms"]
Oct 30 13:24:05.291014 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.290997    5093 scope.go:122] "RemoveContainer" containerID="70fd6e41d90aac4da67c626477432abffab291b4d1c96fa15e5a27cec60480d5"
Oct 30 13:24:05.291407 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:24:05.291377    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70fd6e41d90aac4da67c626477432abffab291b4d1c96fa15e5a27cec60480d5\": container with ID starting with 70fd6e41d90aac4da67c626477432abffab291b4d1c96fa15e5a27cec60480d5 not found: ID does not exist" containerID="70fd6e41d90aac4da67c626477432abffab291b4d1c96fa15e5a27cec60480d5"
Oct 30 13:24:05.291508 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.291416    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70fd6e41d90aac4da67c626477432abffab291b4d1c96fa15e5a27cec60480d5"} err="failed to get container status \"70fd6e41d90aac4da67c626477432abffab291b4d1c96fa15e5a27cec60480d5\": rpc error: code = NotFound desc = could not find container \"70fd6e41d90aac4da67c626477432abffab291b4d1c96fa15e5a27cec60480d5\": container with ID starting with 70fd6e41d90aac4da67c626477432abffab291b4d1c96fa15e5a27cec60480d5 not found: ID does not exist"
Oct 30 13:24:05.294366 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.294343    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-vtzms"]
Oct 30 13:24:05.347143 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.347088    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9"]
Oct 30 13:24:05.353839 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.353804    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-t6td9"]
Oct 30 13:24:05.398243 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.398201    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc"]
Oct 30 13:24:05.403193 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.403162    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-gzskc"]
Oct 30 13:24:05.449117 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.449068    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls"]
Oct 30 13:24:05.452665 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:05.452631    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-sb5ls"]
Oct 30 13:24:06.256139 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:06.256104    5093 generic.go:356] "Generic (PLEG): container finished" podID="58244ace-ebd4-4b98-929b-8396655992a6" containerID="bb40ce33d522380835ac6b8cf3867afdbf64bdf3052b03b320cfa389404fdf81" exitCode=0
Oct 30 13:24:06.256654 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:06.256190    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf" event={"ID":"58244ace-ebd4-4b98-929b-8396655992a6","Type":"ContainerDied","Data":"bb40ce33d522380835ac6b8cf3867afdbf64bdf3052b03b320cfa389404fdf81"}
Oct 30 13:24:06.256727 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:06.256710    5093 scope.go:122] "RemoveContainer" containerID="bb40ce33d522380835ac6b8cf3867afdbf64bdf3052b03b320cfa389404fdf81"
Oct 30 13:24:06.258584 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:06.258554    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:24:06.258726 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:06.258626    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:24:06.258726 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:06.258659    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:24:07.001368 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:07.001328    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b9ce40f-35d8-4041-a510-37f2cd68d2b3" path="/var/lib/kubelet/pods/1b9ce40f-35d8-4041-a510-37f2cd68d2b3/volumes"
Oct 30 13:24:07.001890 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:07.001876    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="616525f4-fcee-4d8e-977e-f2104a46208f" path="/var/lib/kubelet/pods/616525f4-fcee-4d8e-977e-f2104a46208f/volumes"
Oct 30 13:24:07.002388 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:07.002373    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8daf18c1-014e-4c6b-bffb-bb25ceef69f3" path="/var/lib/kubelet/pods/8daf18c1-014e-4c6b-bffb-bb25ceef69f3/volumes"
Oct 30 13:24:07.003161 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:07.003147    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a" path="/var/lib/kubelet/pods/c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a/volumes"
Oct 30 13:24:07.268844 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:07.268762    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf" event={"ID":"58244ace-ebd4-4b98-929b-8396655992a6","Type":"ContainerStarted","Data":"49df42e9ffa51680e1f92bf1ba9bdd1bb46d9c0297c19fe0ff7222e2baea779d"}
Oct 30 13:24:08.137155 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:08.136562    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf"
Oct 30 13:24:08.137155 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:08.136695    5093 patch_prober.go:29] interesting pod/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Oct 30 13:24:08.137155 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:08.136745    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf" podUID="58244ace-ebd4-4b98-929b-8396655992a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Oct 30 13:24:09.137155 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:09.130437    5093 patch_prober.go:29] interesting pod/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Oct 30 13:24:09.137155 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:09.130498    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf" podUID="58244ace-ebd4-4b98-929b-8396655992a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Oct 30 13:24:09.787756 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:09.787716    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk"]
Oct 30 13:24:09.788007 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:09.787980    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk" podUID="f31be4a0-295f-4ed0-901d-f45cee943496" containerName="istio-proxy" containerID="cri-o://e7b90ddc2fd7f3e171c51a7c9a784840409d6376fc3f6f62a5de5b27e85725ad" gracePeriod=30
Oct 30 13:24:09.801552 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:09.801516    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh"]
Oct 30 13:24:09.801784 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:09.801756    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh" podUID="8bf396f4-4bbf-46dd-834e-2621e8c1f6cb" containerName="istio-proxy" containerID="cri-o://8f6368b1b51273a470c57fba46f70113809bb2796759a5a4e169155ac83d1f12" gracePeriod=30
Oct 30 13:24:09.809914 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:09.809249    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn"]
Oct 30 13:24:09.809914 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:09.809557    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn" podUID="874dabda-7e3e-44eb-9768-7ae00d30d22e" containerName="istio-proxy" containerID="cri-o://aee13bbffc9f102ba47b22b3311d8dce547ba0fcb4c925664528bb236704942e" gracePeriod=30
Oct 30 13:24:09.823375 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:09.823344    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr"]
Oct 30 13:24:09.823581 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:09.823558    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr" podUID="b646832b-e0d1-491d-bb54-86546c08081e" containerName="istio-proxy" containerID="cri-o://6fdbb6b17e3c20959727a1be710c4b798ed9302f9edff38cf4694ac31edfa962" gracePeriod=30
Oct 30 13:24:10.148220 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:10.130696    5093 patch_prober.go:29] interesting pod/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Oct 30 13:24:10.148220 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:10.130761    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf" podUID="58244ace-ebd4-4b98-929b-8396655992a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Oct 30 13:24:11.148168 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:11.143624    5093 patch_prober.go:29] interesting pod/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Oct 30 13:24:11.148168 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:11.143710    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf" podUID="58244ace-ebd4-4b98-929b-8396655992a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Oct 30 13:24:12.150154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:12.149475    5093 patch_prober.go:29] interesting pod/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Oct 30 13:24:12.150154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:12.149543    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf" podUID="58244ace-ebd4-4b98-929b-8396655992a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Oct 30 13:24:13.135087 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:13.135048    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf"
Oct 30 13:24:13.135634 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:13.135602    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf"
Oct 30 13:24:13.136254 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:13.136234    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf"
Oct 30 13:24:15.189108 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.189068    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk"
Oct 30 13:24:15.194288 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.194261    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh"
Oct 30 13:24:15.303893 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.303858    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr"
Oct 30 13:24:15.308089 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.308063    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn"
Oct 30 13:24:15.324589 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.324509    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f31be4a0-295f-4ed0-901d-f45cee943496-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f31be4a0-295f-4ed0-901d-f45cee943496-istio-podinfo\") pod \"f31be4a0-295f-4ed0-901d-f45cee943496\" (UID: \"f31be4a0-295f-4ed0-901d-f45cee943496\") "
Oct 30 13:24:15.324589 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.324554    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f31be4a0-295f-4ed0-901d-f45cee943496-kube-api-access-9hsms\" (UniqueName: \"kubernetes.io/projected/f31be4a0-295f-4ed0-901d-f45cee943496-kube-api-access-9hsms\") pod \"f31be4a0-295f-4ed0-901d-f45cee943496\" (UID: \"f31be4a0-295f-4ed0-901d-f45cee943496\") "
Oct 30 13:24:15.324806 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.324596    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-istio-data\") pod \"8bf396f4-4bbf-46dd-834e-2621e8c1f6cb\" (UID: \"8bf396f4-4bbf-46dd-834e-2621e8c1f6cb\") "
Oct 30 13:24:15.324806 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.324627    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-istio-podinfo\") pod \"8bf396f4-4bbf-46dd-834e-2621e8c1f6cb\" (UID: \"8bf396f4-4bbf-46dd-834e-2621e8c1f6cb\") "
Oct 30 13:24:15.324806 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.324647    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-istio-token\" (UniqueName: \"kubernetes.io/projected/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-istio-token\") pod \"8bf396f4-4bbf-46dd-834e-2621e8c1f6cb\" (UID: \"8bf396f4-4bbf-46dd-834e-2621e8c1f6cb\") "
Oct 30 13:24:15.324806 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.324673    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f31be4a0-295f-4ed0-901d-f45cee943496-istio-token\" (UniqueName: \"kubernetes.io/projected/f31be4a0-295f-4ed0-901d-f45cee943496-istio-token\") pod \"f31be4a0-295f-4ed0-901d-f45cee943496\" (UID: \"f31be4a0-295f-4ed0-901d-f45cee943496\") "
Oct 30 13:24:15.324806 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.324703    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f31be4a0-295f-4ed0-901d-f45cee943496-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f31be4a0-295f-4ed0-901d-f45cee943496-credential-socket\") pod \"f31be4a0-295f-4ed0-901d-f45cee943496\" (UID: \"f31be4a0-295f-4ed0-901d-f45cee943496\") "
Oct 30 13:24:15.325070 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.324814    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f31be4a0-295f-4ed0-901d-f45cee943496-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f31be4a0-295f-4ed0-901d-f45cee943496-istio-data\") pod \"f31be4a0-295f-4ed0-901d-f45cee943496\" (UID: \"f31be4a0-295f-4ed0-901d-f45cee943496\") "
Oct 30 13:24:15.325070 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.324907    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f31be4a0-295f-4ed0-901d-f45cee943496-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f31be4a0-295f-4ed0-901d-f45cee943496-istiod-ca-cert\") pod \"f31be4a0-295f-4ed0-901d-f45cee943496\" (UID: \"f31be4a0-295f-4ed0-901d-f45cee943496\") "
Oct 30 13:24:15.325070 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.324910    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-istio-data" pod "8bf396f4-4bbf-46dd-834e-2621e8c1f6cb" (UID: "8bf396f4-4bbf-46dd-834e-2621e8c1f6cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:15.325070 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.324948    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-istiod-ca-cert\") pod \"8bf396f4-4bbf-46dd-834e-2621e8c1f6cb\" (UID: \"8bf396f4-4bbf-46dd-834e-2621e8c1f6cb\") "
Oct 30 13:24:15.325070 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.324985    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f31be4a0-295f-4ed0-901d-f45cee943496-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f31be4a0-295f-4ed0-901d-f45cee943496-workload-certs\") pod \"f31be4a0-295f-4ed0-901d-f45cee943496\" (UID: \"f31be4a0-295f-4ed0-901d-f45cee943496\") "
Oct 30 13:24:15.325070 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.325022    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-istio-envoy\") pod \"8bf396f4-4bbf-46dd-834e-2621e8c1f6cb\" (UID: \"8bf396f4-4bbf-46dd-834e-2621e8c1f6cb\") "
Oct 30 13:24:15.325070 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.325056    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f31be4a0-295f-4ed0-901d-f45cee943496-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f31be4a0-295f-4ed0-901d-f45cee943496-istio-envoy\") pod \"f31be4a0-295f-4ed0-901d-f45cee943496\" (UID: \"f31be4a0-295f-4ed0-901d-f45cee943496\") "
Oct 30 13:24:15.325475 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.325087    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f31be4a0-295f-4ed0-901d-f45cee943496-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f31be4a0-295f-4ed0-901d-f45cee943496-workload-socket\") pod \"f31be4a0-295f-4ed0-901d-f45cee943496\" (UID: \"f31be4a0-295f-4ed0-901d-f45cee943496\") "
Oct 30 13:24:15.325475 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.325112    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f31be4a0-295f-4ed0-901d-f45cee943496-credential-socket" pod "f31be4a0-295f-4ed0-901d-f45cee943496" (UID: "f31be4a0-295f-4ed0-901d-f45cee943496"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:15.325475 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.325147    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-credential-socket\") pod \"8bf396f4-4bbf-46dd-834e-2621e8c1f6cb\" (UID: \"8bf396f4-4bbf-46dd-834e-2621e8c1f6cb\") "
Oct 30 13:24:15.325475 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.325193    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-kube-api-access-gt7sd\" (UniqueName: \"kubernetes.io/projected/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-kube-api-access-gt7sd\") pod \"8bf396f4-4bbf-46dd-834e-2621e8c1f6cb\" (UID: \"8bf396f4-4bbf-46dd-834e-2621e8c1f6cb\") "
Oct 30 13:24:15.325475 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.325349    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-workload-socket\") pod \"8bf396f4-4bbf-46dd-834e-2621e8c1f6cb\" (UID: \"8bf396f4-4bbf-46dd-834e-2621e8c1f6cb\") "
Oct 30 13:24:15.325475 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.325444    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-workload-certs\") pod \"8bf396f4-4bbf-46dd-834e-2621e8c1f6cb\" (UID: \"8bf396f4-4bbf-46dd-834e-2621e8c1f6cb\") "
Oct 30 13:24:15.326339 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.325351    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f31be4a0-295f-4ed0-901d-f45cee943496-istio-data" pod "f31be4a0-295f-4ed0-901d-f45cee943496" (UID: "f31be4a0-295f-4ed0-901d-f45cee943496"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:15.326339 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.325437    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-istiod-ca-cert" pod "8bf396f4-4bbf-46dd-834e-2621e8c1f6cb" (UID: "8bf396f4-4bbf-46dd-834e-2621e8c1f6cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:24:15.326339 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.325779    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f31be4a0-295f-4ed0-901d-f45cee943496-istiod-ca-cert" pod "f31be4a0-295f-4ed0-901d-f45cee943496" (UID: "f31be4a0-295f-4ed0-901d-f45cee943496"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:24:15.326568 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.326441    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f31be4a0-295f-4ed0-901d-f45cee943496-workload-socket" pod "f31be4a0-295f-4ed0-901d-f45cee943496" (UID: "f31be4a0-295f-4ed0-901d-f45cee943496"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:15.327001 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.326724    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-workload-socket" pod "8bf396f4-4bbf-46dd-834e-2621e8c1f6cb" (UID: "8bf396f4-4bbf-46dd-834e-2621e8c1f6cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:15.327001 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.326871    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-credential-socket" pod "8bf396f4-4bbf-46dd-834e-2621e8c1f6cb" (UID: "8bf396f4-4bbf-46dd-834e-2621e8c1f6cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:15.327483 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.327297    5093 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f31be4a0-295f-4ed0-901d-f45cee943496-istiod-ca-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:15.327483 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.327326    5093 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-istiod-ca-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:15.327483 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.327338    5093 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f31be4a0-295f-4ed0-901d-f45cee943496-workload-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:15.327483 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.327335    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f31be4a0-295f-4ed0-901d-f45cee943496-kube-api-access-9hsms" pod "f31be4a0-295f-4ed0-901d-f45cee943496" (UID: "f31be4a0-295f-4ed0-901d-f45cee943496"). InnerVolumeSpecName "kube-api-access-9hsms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:15.327483 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.327351    5093 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-workload-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:15.327483 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.327365    5093 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-istio-data\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:15.327483 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.327379    5093 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f31be4a0-295f-4ed0-901d-f45cee943496-credential-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:15.327483 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.327394    5093 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f31be4a0-295f-4ed0-901d-f45cee943496-istio-data\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:15.327483 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.327461    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f31be4a0-295f-4ed0-901d-f45cee943496-workload-certs" pod "f31be4a0-295f-4ed0-901d-f45cee943496" (UID: "f31be4a0-295f-4ed0-901d-f45cee943496"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:15.328897 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.328865    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-workload-certs" pod "8bf396f4-4bbf-46dd-834e-2621e8c1f6cb" (UID: "8bf396f4-4bbf-46dd-834e-2621e8c1f6cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:15.329282 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.329257    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f31be4a0-295f-4ed0-901d-f45cee943496-istio-envoy" pod "f31be4a0-295f-4ed0-901d-f45cee943496" (UID: "f31be4a0-295f-4ed0-901d-f45cee943496"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:15.329399 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.329372    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-kube-api-access-gt7sd" pod "8bf396f4-4bbf-46dd-834e-2621e8c1f6cb" (UID: "8bf396f4-4bbf-46dd-834e-2621e8c1f6cb"). InnerVolumeSpecName "kube-api-access-gt7sd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:15.329680 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.329637    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f31be4a0-295f-4ed0-901d-f45cee943496-istio-podinfo" pod "f31be4a0-295f-4ed0-901d-f45cee943496" (UID: "f31be4a0-295f-4ed0-901d-f45cee943496"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:24:15.329895 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.329861    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-istio-envoy" pod "8bf396f4-4bbf-46dd-834e-2621e8c1f6cb" (UID: "8bf396f4-4bbf-46dd-834e-2621e8c1f6cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:15.331410 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.330410    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-istio-token" pod "8bf396f4-4bbf-46dd-834e-2621e8c1f6cb" (UID: "8bf396f4-4bbf-46dd-834e-2621e8c1f6cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:15.336739 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.336705    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f31be4a0-295f-4ed0-901d-f45cee943496-istio-token" pod "f31be4a0-295f-4ed0-901d-f45cee943496" (UID: "f31be4a0-295f-4ed0-901d-f45cee943496"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:15.337091 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.337065    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-istio-podinfo" pod "8bf396f4-4bbf-46dd-834e-2621e8c1f6cb" (UID: "8bf396f4-4bbf-46dd-834e-2621e8c1f6cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:24:15.342537 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.342511    5093 generic.go:356] "Generic (PLEG): container finished" podID="874dabda-7e3e-44eb-9768-7ae00d30d22e" containerID="aee13bbffc9f102ba47b22b3311d8dce547ba0fcb4c925664528bb236704942e" exitCode=0
Oct 30 13:24:15.342666 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.342568    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn"
Oct 30 13:24:15.342666 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.342635    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn" event={"ID":"874dabda-7e3e-44eb-9768-7ae00d30d22e","Type":"ContainerDied","Data":"aee13bbffc9f102ba47b22b3311d8dce547ba0fcb4c925664528bb236704942e"}
Oct 30 13:24:15.342773 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.342673    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn" event={"ID":"874dabda-7e3e-44eb-9768-7ae00d30d22e","Type":"ContainerDied","Data":"8869023f376be1454e1c66d067e72f466e6e5ffd2a4a98016396e2c98e8fdb5f"}
Oct 30 13:24:15.342773 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.342695    5093 scope.go:122] "RemoveContainer" containerID="aee13bbffc9f102ba47b22b3311d8dce547ba0fcb4c925664528bb236704942e"
Oct 30 13:24:15.344519 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.344496    5093 generic.go:356] "Generic (PLEG): container finished" podID="b646832b-e0d1-491d-bb54-86546c08081e" containerID="6fdbb6b17e3c20959727a1be710c4b798ed9302f9edff38cf4694ac31edfa962" exitCode=0
Oct 30 13:24:15.344705 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.344681    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr" event={"ID":"b646832b-e0d1-491d-bb54-86546c08081e","Type":"ContainerDied","Data":"6fdbb6b17e3c20959727a1be710c4b798ed9302f9edff38cf4694ac31edfa962"}
Oct 30 13:24:15.344787 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.344734    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr" event={"ID":"b646832b-e0d1-491d-bb54-86546c08081e","Type":"ContainerDied","Data":"6b98599e6077a0563feaa71263c32c04ebc2c9de20794c3a79005d1f2890d1fb"}
Oct 30 13:24:15.344843 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.344811    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr"
Oct 30 13:24:15.346716 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.346695    5093 generic.go:356] "Generic (PLEG): container finished" podID="8bf396f4-4bbf-46dd-834e-2621e8c1f6cb" containerID="8f6368b1b51273a470c57fba46f70113809bb2796759a5a4e169155ac83d1f12" exitCode=0
Oct 30 13:24:15.347214 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.346828    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh"
Oct 30 13:24:15.348432 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.348213    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh" event={"ID":"8bf396f4-4bbf-46dd-834e-2621e8c1f6cb","Type":"ContainerDied","Data":"8f6368b1b51273a470c57fba46f70113809bb2796759a5a4e169155ac83d1f12"}
Oct 30 13:24:15.348432 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.348281    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh" event={"ID":"8bf396f4-4bbf-46dd-834e-2621e8c1f6cb","Type":"ContainerDied","Data":"728fd2161210e75fb87b34835f8b1072811f9e8f0f0dbf3b397b671227f11006"}
Oct 30 13:24:15.375221 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.360070    5093 scope.go:122] "RemoveContainer" containerID="aee13bbffc9f102ba47b22b3311d8dce547ba0fcb4c925664528bb236704942e"
Oct 30 13:24:15.395605 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:24:15.395390    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aee13bbffc9f102ba47b22b3311d8dce547ba0fcb4c925664528bb236704942e\": container with ID starting with aee13bbffc9f102ba47b22b3311d8dce547ba0fcb4c925664528bb236704942e not found: ID does not exist" containerID="aee13bbffc9f102ba47b22b3311d8dce547ba0fcb4c925664528bb236704942e"
Oct 30 13:24:15.395605 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.395446    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aee13bbffc9f102ba47b22b3311d8dce547ba0fcb4c925664528bb236704942e"} err="failed to get container status \"aee13bbffc9f102ba47b22b3311d8dce547ba0fcb4c925664528bb236704942e\": rpc error: code = NotFound desc = could not find container \"aee13bbffc9f102ba47b22b3311d8dce547ba0fcb4c925664528bb236704942e\": container with ID starting with aee13bbffc9f102ba47b22b3311d8dce547ba0fcb4c925664528bb236704942e not found: ID does not exist"
Oct 30 13:24:15.395605 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.395472    5093 scope.go:122] "RemoveContainer" containerID="6fdbb6b17e3c20959727a1be710c4b798ed9302f9edff38cf4694ac31edfa962"
Oct 30 13:24:15.396270 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.396248    5093 generic.go:356] "Generic (PLEG): container finished" podID="f31be4a0-295f-4ed0-901d-f45cee943496" containerID="e7b90ddc2fd7f3e171c51a7c9a784840409d6376fc3f6f62a5de5b27e85725ad" exitCode=0
Oct 30 13:24:15.396475 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.396346    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk"
Oct 30 13:24:15.396682 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.396367    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk" event={"ID":"f31be4a0-295f-4ed0-901d-f45cee943496","Type":"ContainerDied","Data":"e7b90ddc2fd7f3e171c51a7c9a784840409d6376fc3f6f62a5de5b27e85725ad"}
Oct 30 13:24:15.396762 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.396742    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk" event={"ID":"f31be4a0-295f-4ed0-901d-f45cee943496","Type":"ContainerDied","Data":"4a53e2945cda948c91a194a60e64d665686213e085aa11b39f0c4955445833ff"}
Oct 30 13:24:15.413264 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.413147    5093 scope.go:122] "RemoveContainer" containerID="6fdbb6b17e3c20959727a1be710c4b798ed9302f9edff38cf4694ac31edfa962"
Oct 30 13:24:15.413681 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:24:15.413653    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fdbb6b17e3c20959727a1be710c4b798ed9302f9edff38cf4694ac31edfa962\": container with ID starting with 6fdbb6b17e3c20959727a1be710c4b798ed9302f9edff38cf4694ac31edfa962 not found: ID does not exist" containerID="6fdbb6b17e3c20959727a1be710c4b798ed9302f9edff38cf4694ac31edfa962"
Oct 30 13:24:15.413772 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.413696    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fdbb6b17e3c20959727a1be710c4b798ed9302f9edff38cf4694ac31edfa962"} err="failed to get container status \"6fdbb6b17e3c20959727a1be710c4b798ed9302f9edff38cf4694ac31edfa962\": rpc error: code = NotFound desc = could not find container \"6fdbb6b17e3c20959727a1be710c4b798ed9302f9edff38cf4694ac31edfa962\": container with ID starting with 6fdbb6b17e3c20959727a1be710c4b798ed9302f9edff38cf4694ac31edfa962 not found: ID does not exist"
Oct 30 13:24:15.413772 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.413722    5093 scope.go:122] "RemoveContainer" containerID="8f6368b1b51273a470c57fba46f70113809bb2796759a5a4e169155ac83d1f12"
Oct 30 13:24:15.425985 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.425954    5093 scope.go:122] "RemoveContainer" containerID="8f6368b1b51273a470c57fba46f70113809bb2796759a5a4e169155ac83d1f12"
Oct 30 13:24:15.428006 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.427979    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/874dabda-7e3e-44eb-9768-7ae00d30d22e-istio-token\" (UniqueName: \"kubernetes.io/projected/874dabda-7e3e-44eb-9768-7ae00d30d22e-istio-token\") pod \"874dabda-7e3e-44eb-9768-7ae00d30d22e\" (UID: \"874dabda-7e3e-44eb-9768-7ae00d30d22e\") "
Oct 30 13:24:15.428171 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.428033    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/874dabda-7e3e-44eb-9768-7ae00d30d22e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/874dabda-7e3e-44eb-9768-7ae00d30d22e-istio-envoy\") pod \"874dabda-7e3e-44eb-9768-7ae00d30d22e\" (UID: \"874dabda-7e3e-44eb-9768-7ae00d30d22e\") "
Oct 30 13:24:15.428171 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.428116    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b646832b-e0d1-491d-bb54-86546c08081e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b646832b-e0d1-491d-bb54-86546c08081e-credential-socket\") pod \"b646832b-e0d1-491d-bb54-86546c08081e\" (UID: \"b646832b-e0d1-491d-bb54-86546c08081e\") "
Oct 30 13:24:15.432531 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.432506    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b646832b-e0d1-491d-bb54-86546c08081e-kube-api-access-5kkrp\" (UniqueName: \"kubernetes.io/projected/b646832b-e0d1-491d-bb54-86546c08081e-kube-api-access-5kkrp\") pod \"b646832b-e0d1-491d-bb54-86546c08081e\" (UID: \"b646832b-e0d1-491d-bb54-86546c08081e\") "
Oct 30 13:24:15.432643 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.432563    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b646832b-e0d1-491d-bb54-86546c08081e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b646832b-e0d1-491d-bb54-86546c08081e-istio-data\") pod \"b646832b-e0d1-491d-bb54-86546c08081e\" (UID: \"b646832b-e0d1-491d-bb54-86546c08081e\") "
Oct 30 13:24:15.432643 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.432591    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b646832b-e0d1-491d-bb54-86546c08081e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b646832b-e0d1-491d-bb54-86546c08081e-istiod-ca-cert\") pod \"b646832b-e0d1-491d-bb54-86546c08081e\" (UID: \"b646832b-e0d1-491d-bb54-86546c08081e\") "
Oct 30 13:24:15.432643 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.432630    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/874dabda-7e3e-44eb-9768-7ae00d30d22e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/874dabda-7e3e-44eb-9768-7ae00d30d22e-credential-socket\") pod \"874dabda-7e3e-44eb-9768-7ae00d30d22e\" (UID: \"874dabda-7e3e-44eb-9768-7ae00d30d22e\") "
Oct 30 13:24:15.432872 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.432672    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b646832b-e0d1-491d-bb54-86546c08081e-istio-token\" (UniqueName: \"kubernetes.io/projected/b646832b-e0d1-491d-bb54-86546c08081e-istio-token\") pod \"b646832b-e0d1-491d-bb54-86546c08081e\" (UID: \"b646832b-e0d1-491d-bb54-86546c08081e\") "
Oct 30 13:24:15.432872 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.432733    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/874dabda-7e3e-44eb-9768-7ae00d30d22e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/874dabda-7e3e-44eb-9768-7ae00d30d22e-istiod-ca-cert\") pod \"874dabda-7e3e-44eb-9768-7ae00d30d22e\" (UID: \"874dabda-7e3e-44eb-9768-7ae00d30d22e\") "
Oct 30 13:24:15.432872 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.428721    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b646832b-e0d1-491d-bb54-86546c08081e-credential-socket" pod "b646832b-e0d1-491d-bb54-86546c08081e" (UID: "b646832b-e0d1-491d-bb54-86546c08081e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:15.432872 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.432828    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b646832b-e0d1-491d-bb54-86546c08081e-istio-data" pod "b646832b-e0d1-491d-bb54-86546c08081e" (UID: "b646832b-e0d1-491d-bb54-86546c08081e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:15.432872 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:24:15.428315    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f6368b1b51273a470c57fba46f70113809bb2796759a5a4e169155ac83d1f12\": container with ID starting with 8f6368b1b51273a470c57fba46f70113809bb2796759a5a4e169155ac83d1f12 not found: ID does not exist" containerID="8f6368b1b51273a470c57fba46f70113809bb2796759a5a4e169155ac83d1f12"
Oct 30 13:24:15.433034 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.432910    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f6368b1b51273a470c57fba46f70113809bb2796759a5a4e169155ac83d1f12"} err="failed to get container status \"8f6368b1b51273a470c57fba46f70113809bb2796759a5a4e169155ac83d1f12\": rpc error: code = NotFound desc = could not find container \"8f6368b1b51273a470c57fba46f70113809bb2796759a5a4e169155ac83d1f12\": container with ID starting with 8f6368b1b51273a470c57fba46f70113809bb2796759a5a4e169155ac83d1f12 not found: ID does not exist"
Oct 30 13:24:15.433034 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.432939    5093 scope.go:122] "RemoveContainer" containerID="e7b90ddc2fd7f3e171c51a7c9a784840409d6376fc3f6f62a5de5b27e85725ad"
Oct 30 13:24:15.433868 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.433839    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/874dabda-7e3e-44eb-9768-7ae00d30d22e-istiod-ca-cert" pod "874dabda-7e3e-44eb-9768-7ae00d30d22e" (UID: "874dabda-7e3e-44eb-9768-7ae00d30d22e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:24:15.433973 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.433887    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/874dabda-7e3e-44eb-9768-7ae00d30d22e-credential-socket" pod "874dabda-7e3e-44eb-9768-7ae00d30d22e" (UID: "874dabda-7e3e-44eb-9768-7ae00d30d22e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:15.434045 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.434026    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b646832b-e0d1-491d-bb54-86546c08081e-istiod-ca-cert" pod "b646832b-e0d1-491d-bb54-86546c08081e" (UID: "b646832b-e0d1-491d-bb54-86546c08081e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:24:15.434415 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.434331    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/874dabda-7e3e-44eb-9768-7ae00d30d22e-kube-api-access-94hq4\" (UniqueName: \"kubernetes.io/projected/874dabda-7e3e-44eb-9768-7ae00d30d22e-kube-api-access-94hq4\") pod \"874dabda-7e3e-44eb-9768-7ae00d30d22e\" (UID: \"874dabda-7e3e-44eb-9768-7ae00d30d22e\") "
Oct 30 13:24:15.434777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.434757    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b646832b-e0d1-491d-bb54-86546c08081e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b646832b-e0d1-491d-bb54-86546c08081e-workload-certs\") pod \"b646832b-e0d1-491d-bb54-86546c08081e\" (UID: \"b646832b-e0d1-491d-bb54-86546c08081e\") "
Oct 30 13:24:15.434881 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.434830    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b646832b-e0d1-491d-bb54-86546c08081e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b646832b-e0d1-491d-bb54-86546c08081e-istio-envoy\") pod \"b646832b-e0d1-491d-bb54-86546c08081e\" (UID: \"b646832b-e0d1-491d-bb54-86546c08081e\") "
Oct 30 13:24:15.434950 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.434909    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b646832b-e0d1-491d-bb54-86546c08081e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b646832b-e0d1-491d-bb54-86546c08081e-istio-podinfo\") pod \"b646832b-e0d1-491d-bb54-86546c08081e\" (UID: \"b646832b-e0d1-491d-bb54-86546c08081e\") "
Oct 30 13:24:15.435005 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.434969    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/874dabda-7e3e-44eb-9768-7ae00d30d22e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/874dabda-7e3e-44eb-9768-7ae00d30d22e-workload-certs\") pod \"874dabda-7e3e-44eb-9768-7ae00d30d22e\" (UID: \"874dabda-7e3e-44eb-9768-7ae00d30d22e\") "
Oct 30 13:24:15.435053 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.435015    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/874dabda-7e3e-44eb-9768-7ae00d30d22e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/874dabda-7e3e-44eb-9768-7ae00d30d22e-istio-podinfo\") pod \"874dabda-7e3e-44eb-9768-7ae00d30d22e\" (UID: \"874dabda-7e3e-44eb-9768-7ae00d30d22e\") "
Oct 30 13:24:15.435106 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.435086    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/874dabda-7e3e-44eb-9768-7ae00d30d22e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/874dabda-7e3e-44eb-9768-7ae00d30d22e-istio-data\") pod \"874dabda-7e3e-44eb-9768-7ae00d30d22e\" (UID: \"874dabda-7e3e-44eb-9768-7ae00d30d22e\") "
Oct 30 13:24:15.435266 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.435190    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b646832b-e0d1-491d-bb54-86546c08081e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b646832b-e0d1-491d-bb54-86546c08081e-workload-socket\") pod \"b646832b-e0d1-491d-bb54-86546c08081e\" (UID: \"b646832b-e0d1-491d-bb54-86546c08081e\") "
Oct 30 13:24:15.435266 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.435219    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/874dabda-7e3e-44eb-9768-7ae00d30d22e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/874dabda-7e3e-44eb-9768-7ae00d30d22e-workload-socket\") pod \"874dabda-7e3e-44eb-9768-7ae00d30d22e\" (UID: \"874dabda-7e3e-44eb-9768-7ae00d30d22e\") "
Oct 30 13:24:15.436103 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.435875    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/874dabda-7e3e-44eb-9768-7ae00d30d22e-workload-certs" pod "874dabda-7e3e-44eb-9768-7ae00d30d22e" (UID: "874dabda-7e3e-44eb-9768-7ae00d30d22e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:15.436103 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.435942    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b646832b-e0d1-491d-bb54-86546c08081e-kube-api-access-5kkrp" pod "b646832b-e0d1-491d-bb54-86546c08081e" (UID: "b646832b-e0d1-491d-bb54-86546c08081e"). InnerVolumeSpecName "kube-api-access-5kkrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:15.436103 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.436061    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b646832b-e0d1-491d-bb54-86546c08081e-workload-certs" pod "b646832b-e0d1-491d-bb54-86546c08081e" (UID: "b646832b-e0d1-491d-bb54-86546c08081e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:15.436398 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.436100    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/874dabda-7e3e-44eb-9768-7ae00d30d22e-istio-data" pod "874dabda-7e3e-44eb-9768-7ae00d30d22e" (UID: "874dabda-7e3e-44eb-9768-7ae00d30d22e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:15.436398 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.436207    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b646832b-e0d1-491d-bb54-86546c08081e-workload-socket" pod "b646832b-e0d1-491d-bb54-86546c08081e" (UID: "b646832b-e0d1-491d-bb54-86546c08081e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:15.437148 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.436717    5093 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f31be4a0-295f-4ed0-901d-f45cee943496-workload-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:15.437148 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.436778    5093 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-istio-envoy\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:15.437148 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.436794    5093 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f31be4a0-295f-4ed0-901d-f45cee943496-istio-envoy\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:15.437148 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.436800    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/874dabda-7e3e-44eb-9768-7ae00d30d22e-workload-socket" pod "874dabda-7e3e-44eb-9768-7ae00d30d22e" (UID: "874dabda-7e3e-44eb-9768-7ae00d30d22e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:15.437148 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.436808    5093 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-credential-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:15.437148 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.436823    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt7sd\" (UniqueName: \"kubernetes.io/projected/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-kube-api-access-gt7sd\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:15.437148 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.436842    5093 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b646832b-e0d1-491d-bb54-86546c08081e-credential-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:15.437148 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.436861    5093 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b646832b-e0d1-491d-bb54-86546c08081e-istio-data\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:15.437148 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.436874    5093 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b646832b-e0d1-491d-bb54-86546c08081e-istiod-ca-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:15.437148 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.436887    5093 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/874dabda-7e3e-44eb-9768-7ae00d30d22e-credential-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:15.437148 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.436902    5093 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-workload-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:15.437148 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.436916    5093 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f31be4a0-295f-4ed0-901d-f45cee943496-istio-podinfo\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:15.437148 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.436929    5093 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/874dabda-7e3e-44eb-9768-7ae00d30d22e-istiod-ca-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:15.437148 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.436943    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hsms\" (UniqueName: \"kubernetes.io/projected/f31be4a0-295f-4ed0-901d-f45cee943496-kube-api-access-9hsms\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:15.437148 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.436955    5093 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-istio-podinfo\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:15.437148 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.436970    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:15.437148 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.436982    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f31be4a0-295f-4ed0-901d-f45cee943496-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:15.439438 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.439267    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b646832b-e0d1-491d-bb54-86546c08081e-istio-envoy" pod "b646832b-e0d1-491d-bb54-86546c08081e" (UID: "b646832b-e0d1-491d-bb54-86546c08081e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:15.441582 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.441541    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/874dabda-7e3e-44eb-9768-7ae00d30d22e-istio-envoy" pod "874dabda-7e3e-44eb-9768-7ae00d30d22e" (UID: "874dabda-7e3e-44eb-9768-7ae00d30d22e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:15.441776 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.441750    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/874dabda-7e3e-44eb-9768-7ae00d30d22e-istio-podinfo" pod "874dabda-7e3e-44eb-9768-7ae00d30d22e" (UID: "874dabda-7e3e-44eb-9768-7ae00d30d22e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:24:15.443339 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.443311    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b646832b-e0d1-491d-bb54-86546c08081e-istio-podinfo" pod "b646832b-e0d1-491d-bb54-86546c08081e" (UID: "b646832b-e0d1-491d-bb54-86546c08081e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:24:15.445623 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.445597    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b646832b-e0d1-491d-bb54-86546c08081e-istio-token" pod "b646832b-e0d1-491d-bb54-86546c08081e" (UID: "b646832b-e0d1-491d-bb54-86546c08081e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:15.446963 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.446937    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/874dabda-7e3e-44eb-9768-7ae00d30d22e-kube-api-access-94hq4" pod "874dabda-7e3e-44eb-9768-7ae00d30d22e" (UID: "874dabda-7e3e-44eb-9768-7ae00d30d22e"). InnerVolumeSpecName "kube-api-access-94hq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:15.452302 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.452282    5093 scope.go:122] "RemoveContainer" containerID="e7b90ddc2fd7f3e171c51a7c9a784840409d6376fc3f6f62a5de5b27e85725ad"
Oct 30 13:24:15.452659 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:24:15.452639    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7b90ddc2fd7f3e171c51a7c9a784840409d6376fc3f6f62a5de5b27e85725ad\": container with ID starting with e7b90ddc2fd7f3e171c51a7c9a784840409d6376fc3f6f62a5de5b27e85725ad not found: ID does not exist" containerID="e7b90ddc2fd7f3e171c51a7c9a784840409d6376fc3f6f62a5de5b27e85725ad"
Oct 30 13:24:15.452750 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.452673    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7b90ddc2fd7f3e171c51a7c9a784840409d6376fc3f6f62a5de5b27e85725ad"} err="failed to get container status \"e7b90ddc2fd7f3e171c51a7c9a784840409d6376fc3f6f62a5de5b27e85725ad\": rpc error: code = NotFound desc = could not find container \"e7b90ddc2fd7f3e171c51a7c9a784840409d6376fc3f6f62a5de5b27e85725ad\": container with ID starting with e7b90ddc2fd7f3e171c51a7c9a784840409d6376fc3f6f62a5de5b27e85725ad not found: ID does not exist"
Oct 30 13:24:15.456644 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.456622    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/874dabda-7e3e-44eb-9768-7ae00d30d22e-istio-token" pod "874dabda-7e3e-44eb-9768-7ae00d30d22e" (UID: "874dabda-7e3e-44eb-9768-7ae00d30d22e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:15.538471 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.538428    5093 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b646832b-e0d1-491d-bb54-86546c08081e-workload-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:15.538471 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.538468    5093 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/874dabda-7e3e-44eb-9768-7ae00d30d22e-workload-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:15.538732 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.538484    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/874dabda-7e3e-44eb-9768-7ae00d30d22e-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:15.538732 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.538498    5093 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/874dabda-7e3e-44eb-9768-7ae00d30d22e-istio-envoy\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:15.538732 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.538512    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kkrp\" (UniqueName: \"kubernetes.io/projected/b646832b-e0d1-491d-bb54-86546c08081e-kube-api-access-5kkrp\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:15.538732 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.538527    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b646832b-e0d1-491d-bb54-86546c08081e-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:15.538732 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.538544    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94hq4\" (UniqueName: \"kubernetes.io/projected/874dabda-7e3e-44eb-9768-7ae00d30d22e-kube-api-access-94hq4\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:15.538732 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.538557    5093 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b646832b-e0d1-491d-bb54-86546c08081e-workload-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:15.538732 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.538569    5093 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b646832b-e0d1-491d-bb54-86546c08081e-istio-envoy\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:15.538732 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.538581    5093 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b646832b-e0d1-491d-bb54-86546c08081e-istio-podinfo\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:15.538732 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.538595    5093 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/874dabda-7e3e-44eb-9768-7ae00d30d22e-workload-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:15.538732 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.538607    5093 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/874dabda-7e3e-44eb-9768-7ae00d30d22e-istio-podinfo\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:15.538732 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.538619    5093 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/874dabda-7e3e-44eb-9768-7ae00d30d22e-istio-data\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:24:15.552768 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.552729    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh"]
Oct 30 13:24:15.558714 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.558688    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-h2xxh"]
Oct 30 13:24:15.604523 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.604414    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk"]
Oct 30 13:24:15.607994 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.607969    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-48fmk"]
Oct 30 13:24:15.698413 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.698380    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn"]
Oct 30 13:24:15.702272 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.702248    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-npxvn"]
Oct 30 13:24:15.737329 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.737295    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr"]
Oct 30 13:24:15.747835 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:15.741925    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-8cpfr"]
Oct 30 13:24:17.002418 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:17.002385    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="874dabda-7e3e-44eb-9768-7ae00d30d22e" path="/var/lib/kubelet/pods/874dabda-7e3e-44eb-9768-7ae00d30d22e/volumes"
Oct 30 13:24:17.002880 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:17.002866    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bf396f4-4bbf-46dd-834e-2621e8c1f6cb" path="/var/lib/kubelet/pods/8bf396f4-4bbf-46dd-834e-2621e8c1f6cb/volumes"
Oct 30 13:24:17.003326 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:17.003313    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b646832b-e0d1-491d-bb54-86546c08081e" path="/var/lib/kubelet/pods/b646832b-e0d1-491d-bb54-86546c08081e/volumes"
Oct 30 13:24:17.004096 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:17.004080    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f31be4a0-295f-4ed0-901d-f45cee943496" path="/var/lib/kubelet/pods/f31be4a0-295f-4ed0-901d-f45cee943496/volumes"
Oct 30 13:24:19.334352 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:19.334313    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e3e617-d9869f767-bmr4t"]
Oct 30 13:24:19.334950 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:19.334934    5093 cpu_manager.go:414] "containerMap: removing container" podUID="6e9f2890-8091-4bd7-8471-9c7d97081bd3" containerName="istio-proxy"
Oct 30 13:24:19.335008 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:19.334951    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e9f2890-8091-4bd7-8471-9c7d97081bd3" containerName="istio-proxy"
Oct 30 13:24:19.335008 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:19.334963    5093 cpu_manager.go:414] "containerMap: removing container" podUID="8bf396f4-4bbf-46dd-834e-2621e8c1f6cb" containerName="istio-proxy"
Oct 30 13:24:19.335008 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:19.334968    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bf396f4-4bbf-46dd-834e-2621e8c1f6cb" containerName="istio-proxy"
Oct 30 13:24:19.335008 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:19.334986    5093 cpu_manager.go:414] "containerMap: removing container" podUID="616525f4-fcee-4d8e-977e-f2104a46208f" containerName="istio-proxy"
Oct 30 13:24:19.335008 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:19.334992    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="616525f4-fcee-4d8e-977e-f2104a46208f" containerName="istio-proxy"
Oct 30 13:24:19.335008 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:19.335001    5093 cpu_manager.go:414] "containerMap: removing container" podUID="1b9ce40f-35d8-4041-a510-37f2cd68d2b3" containerName="istio-proxy"
Oct 30 13:24:19.335008 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:19.335006    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b9ce40f-35d8-4041-a510-37f2cd68d2b3" containerName="istio-proxy"
Oct 30 13:24:19.335243 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:19.335018    5093 cpu_manager.go:414] "containerMap: removing container" podUID="b646832b-e0d1-491d-bb54-86546c08081e" containerName="istio-proxy"
Oct 30 13:24:19.335243 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:19.335023    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b646832b-e0d1-491d-bb54-86546c08081e" containerName="istio-proxy"
Oct 30 13:24:19.335243 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:19.335039    5093 cpu_manager.go:414] "containerMap: removing container" podUID="8daf18c1-014e-4c6b-bffb-bb25ceef69f3" containerName="istio-proxy"
Oct 30 13:24:19.335243 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:19.335043    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8daf18c1-014e-4c6b-bffb-bb25ceef69f3" containerName="istio-proxy"
Oct 30 13:24:19.335243 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:19.335054    5093 cpu_manager.go:414] "containerMap: removing container" podUID="f31be4a0-295f-4ed0-901d-f45cee943496" containerName="istio-proxy"
Oct 30 13:24:19.335243 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:19.335059    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f31be4a0-295f-4ed0-901d-f45cee943496" containerName="istio-proxy"
Oct 30 13:24:19.335243 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:19.335065    5093 cpu_manager.go:414] "containerMap: removing container" podUID="874dabda-7e3e-44eb-9768-7ae00d30d22e" containerName="istio-proxy"
Oct 30 13:24:19.335243 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:19.335069    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="874dabda-7e3e-44eb-9768-7ae00d30d22e" containerName="istio-proxy"
Oct 30 13:24:19.335243 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:19.335077    5093 cpu_manager.go:414] "containerMap: removing container" podUID="c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a" containerName="istio-proxy"
Oct 30 13:24:19.335243 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:19.335081    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a" containerName="istio-proxy"
Oct 30 13:24:19.335243 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:19.335177    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="616525f4-fcee-4d8e-977e-f2104a46208f" containerName="istio-proxy"
Oct 30 13:24:19.335243 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:19.335187    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b9ce40f-35d8-4041-a510-37f2cd68d2b3" containerName="istio-proxy"
Oct 30 13:24:19.335243 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:19.335193    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e9f2890-8091-4bd7-8471-9c7d97081bd3" containerName="istio-proxy"
Oct 30 13:24:19.335243 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:19.335200    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="b646832b-e0d1-491d-bb54-86546c08081e" containerName="istio-proxy"
Oct 30 13:24:19.335243 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:19.335208    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="8daf18c1-014e-4c6b-bffb-bb25ceef69f3" containerName="istio-proxy"
Oct 30 13:24:19.335243 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:19.335215    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="f31be4a0-295f-4ed0-901d-f45cee943496" containerName="istio-proxy"
Oct 30 13:24:19.335243 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:19.335221    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="874dabda-7e3e-44eb-9768-7ae00d30d22e" containerName="istio-proxy"
Oct 30 13:24:19.335243 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:19.335225    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="c49a3d68-55ef-4c8a-8d1e-e7eb1ea7600a" containerName="istio-proxy"
Oct 30 13:24:19.335243 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:19.335234    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bf396f4-4bbf-46dd-834e-2621e8c1f6cb" containerName="istio-proxy"
Oct 30 13:24:19.338482 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:19.338465    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e3e617-d9869f767-bmr4t"
Oct 30 13:24:19.364567 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:19.364537    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-e3e617-serving-cert\""
Oct 30 13:24:19.375944 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:19.374924    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e3e617-d9869f767-bmr4t"]
Oct 30 13:24:19.493453 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:19.493418    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aadc22cb-ee34-43f9-9071-2d9f86044235-openshift-service-ca\") pod \"maas-api-e2e-api-b-e3e617-d9869f767-bmr4t\" (UID: \"aadc22cb-ee34-43f9-9071-2d9f86044235\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e3e617-d9869f767-bmr4t"
Oct 30 13:24:19.493453 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:19.493455    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aadc22cb-ee34-43f9-9071-2d9f86044235-metrics-certs\") pod \"maas-api-e2e-api-b-e3e617-d9869f767-bmr4t\" (UID: \"aadc22cb-ee34-43f9-9071-2d9f86044235\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e3e617-d9869f767-bmr4t"
Oct 30 13:24:19.493684 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:19.493630    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwkdv\" (UniqueName: \"kubernetes.io/projected/aadc22cb-ee34-43f9-9071-2d9f86044235-kube-api-access-qwkdv\") pod \"maas-api-e2e-api-b-e3e617-d9869f767-bmr4t\" (UID: \"aadc22cb-ee34-43f9-9071-2d9f86044235\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e3e617-d9869f767-bmr4t"
Oct 30 13:24:19.493684 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:19.493678    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aadc22cb-ee34-43f9-9071-2d9f86044235-maas-api-tls\") pod \"maas-api-e2e-api-b-e3e617-d9869f767-bmr4t\" (UID: \"aadc22cb-ee34-43f9-9071-2d9f86044235\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e3e617-d9869f767-bmr4t"
Oct 30 13:24:19.595713 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:19.595602    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwkdv\" (UniqueName: \"kubernetes.io/projected/aadc22cb-ee34-43f9-9071-2d9f86044235-kube-api-access-qwkdv\") pod \"maas-api-e2e-api-b-e3e617-d9869f767-bmr4t\" (UID: \"aadc22cb-ee34-43f9-9071-2d9f86044235\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e3e617-d9869f767-bmr4t"
Oct 30 13:24:19.595713 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:19.595677    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aadc22cb-ee34-43f9-9071-2d9f86044235-maas-api-tls\") pod \"maas-api-e2e-api-b-e3e617-d9869f767-bmr4t\" (UID: \"aadc22cb-ee34-43f9-9071-2d9f86044235\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e3e617-d9869f767-bmr4t"
Oct 30 13:24:19.595954 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:19.595834    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aadc22cb-ee34-43f9-9071-2d9f86044235-openshift-service-ca\") pod \"maas-api-e2e-api-b-e3e617-d9869f767-bmr4t\" (UID: \"aadc22cb-ee34-43f9-9071-2d9f86044235\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e3e617-d9869f767-bmr4t"
Oct 30 13:24:19.595954 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:19.595857    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aadc22cb-ee34-43f9-9071-2d9f86044235-metrics-certs\") pod \"maas-api-e2e-api-b-e3e617-d9869f767-bmr4t\" (UID: \"aadc22cb-ee34-43f9-9071-2d9f86044235\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e3e617-d9869f767-bmr4t"
Oct 30 13:24:19.596811 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:19.596781    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aadc22cb-ee34-43f9-9071-2d9f86044235-openshift-service-ca\") pod \"maas-api-e2e-api-b-e3e617-d9869f767-bmr4t\" (UID: \"aadc22cb-ee34-43f9-9071-2d9f86044235\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e3e617-d9869f767-bmr4t"
Oct 30 13:24:19.599184 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:19.599097    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aadc22cb-ee34-43f9-9071-2d9f86044235-maas-api-tls\") pod \"maas-api-e2e-api-b-e3e617-d9869f767-bmr4t\" (UID: \"aadc22cb-ee34-43f9-9071-2d9f86044235\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e3e617-d9869f767-bmr4t"
Oct 30 13:24:19.599503 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:19.599487    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aadc22cb-ee34-43f9-9071-2d9f86044235-metrics-certs\") pod \"maas-api-e2e-api-b-e3e617-d9869f767-bmr4t\" (UID: \"aadc22cb-ee34-43f9-9071-2d9f86044235\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e3e617-d9869f767-bmr4t"
Oct 30 13:24:19.616084 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:19.616052    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwkdv\" (UniqueName: \"kubernetes.io/projected/aadc22cb-ee34-43f9-9071-2d9f86044235-kube-api-access-qwkdv\") pod \"maas-api-e2e-api-b-e3e617-d9869f767-bmr4t\" (UID: \"aadc22cb-ee34-43f9-9071-2d9f86044235\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e3e617-d9869f767-bmr4t"
Oct 30 13:24:19.653299 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:19.653260    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e3e617-d9869f767-bmr4t"
Oct 30 13:24:19.959692 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:19.959657    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e3e617-d9869f767-bmr4t"]
Oct 30 13:24:19.961206 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:24:19.961173    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaadc22cb_ee34_43f9_9071_2d9f86044235.slice/crio-bc287cabf015f51322dc31e30fcd9a41dd68dd70e46df78ca59795788f5b5d59 WatchSource:0}: Error finding container bc287cabf015f51322dc31e30fcd9a41dd68dd70e46df78ca59795788f5b5d59: Status 404 returned error can't find the container with id bc287cabf015f51322dc31e30fcd9a41dd68dd70e46df78ca59795788f5b5d59
Oct 30 13:24:19.962827 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:19.962798    5093 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 13:24:20.452424 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:20.452385    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e3e617-d9869f767-bmr4t" event={"ID":"aadc22cb-ee34-43f9-9071-2d9f86044235","Type":"ContainerStarted","Data":"afd87389dce0fea565a85a307fa2f03e83d13f15992491ba8192af962aba18db"}
Oct 30 13:24:20.452901 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:20.452451    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e3e617-d9869f767-bmr4t" event={"ID":"aadc22cb-ee34-43f9-9071-2d9f86044235","Type":"ContainerStarted","Data":"bc287cabf015f51322dc31e30fcd9a41dd68dd70e46df78ca59795788f5b5d59"}
Oct 30 13:24:20.452901 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:20.452481    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e3e617-d9869f767-bmr4t"
Oct 30 13:24:20.529221 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:20.518454    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e3e617-d9869f767-bmr4t" podStartSLOduration=1.176083112 podStartE2EDuration="1.518435175s" totalImagesPullingTime="342.352063ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:24:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:24:20.501601064 +0000 UTC m=+1279.648868305" watchObservedRunningTime="2026-10-30 13:24:20.518435175 +0000 UTC m=+1279.665702417"
Oct 30 13:24:21.169096 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:24:21.169051    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:24:25.025253 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:25.025221    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5cee02-7c59cdc9fb-6b47j"]
Oct 30 13:24:25.025904 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:25.025888    5093 cpu_manager.go:414] "containerMap: removing container" podUID="6e9f2890-8091-4bd7-8471-9c7d97081bd3" containerName="istio-proxy"
Oct 30 13:24:25.025966 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:25.025906    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e9f2890-8091-4bd7-8471-9c7d97081bd3" containerName="istio-proxy"
Oct 30 13:24:25.026023 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:25.026013    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e9f2890-8091-4bd7-8471-9c7d97081bd3" containerName="istio-proxy"
Oct 30 13:24:25.029810 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:25.029791    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5cee02-7c59cdc9fb-6b47j"
Oct 30 13:24:25.050648 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:25.050612    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5cee02-7c59cdc9fb-6b47j"]
Oct 30 13:24:25.065015 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:25.064980    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-5cee02-serving-cert\""
Oct 30 13:24:25.184121 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:25.184082    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee1d1f98-eee4-47c2-8ef7-5204fa1d787f-metrics-certs\") pod \"maas-api-e2e-praxis-a-5cee02-7c59cdc9fb-6b47j\" (UID: \"ee1d1f98-eee4-47c2-8ef7-5204fa1d787f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5cee02-7c59cdc9fb-6b47j"
Oct 30 13:24:25.184323 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:25.184163    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee1d1f98-eee4-47c2-8ef7-5204fa1d787f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-5cee02-7c59cdc9fb-6b47j\" (UID: \"ee1d1f98-eee4-47c2-8ef7-5204fa1d787f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5cee02-7c59cdc9fb-6b47j"
Oct 30 13:24:25.184371 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:25.184340    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnppw\" (UniqueName: \"kubernetes.io/projected/ee1d1f98-eee4-47c2-8ef7-5204fa1d787f-kube-api-access-gnppw\") pod \"maas-api-e2e-praxis-a-5cee02-7c59cdc9fb-6b47j\" (UID: \"ee1d1f98-eee4-47c2-8ef7-5204fa1d787f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5cee02-7c59cdc9fb-6b47j"
Oct 30 13:24:25.184408 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:25.184382    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee1d1f98-eee4-47c2-8ef7-5204fa1d787f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-5cee02-7c59cdc9fb-6b47j\" (UID: \"ee1d1f98-eee4-47c2-8ef7-5204fa1d787f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5cee02-7c59cdc9fb-6b47j"
Oct 30 13:24:25.286630 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:25.286522    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnppw\" (UniqueName: \"kubernetes.io/projected/ee1d1f98-eee4-47c2-8ef7-5204fa1d787f-kube-api-access-gnppw\") pod \"maas-api-e2e-praxis-a-5cee02-7c59cdc9fb-6b47j\" (UID: \"ee1d1f98-eee4-47c2-8ef7-5204fa1d787f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5cee02-7c59cdc9fb-6b47j"
Oct 30 13:24:25.286630 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:25.286582    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee1d1f98-eee4-47c2-8ef7-5204fa1d787f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-5cee02-7c59cdc9fb-6b47j\" (UID: \"ee1d1f98-eee4-47c2-8ef7-5204fa1d787f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5cee02-7c59cdc9fb-6b47j"
Oct 30 13:24:25.286845 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:25.286717    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee1d1f98-eee4-47c2-8ef7-5204fa1d787f-metrics-certs\") pod \"maas-api-e2e-praxis-a-5cee02-7c59cdc9fb-6b47j\" (UID: \"ee1d1f98-eee4-47c2-8ef7-5204fa1d787f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5cee02-7c59cdc9fb-6b47j"
Oct 30 13:24:25.286845 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:25.286753    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee1d1f98-eee4-47c2-8ef7-5204fa1d787f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-5cee02-7c59cdc9fb-6b47j\" (UID: \"ee1d1f98-eee4-47c2-8ef7-5204fa1d787f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5cee02-7c59cdc9fb-6b47j"
Oct 30 13:24:25.287661 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:25.287629    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee1d1f98-eee4-47c2-8ef7-5204fa1d787f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-5cee02-7c59cdc9fb-6b47j\" (UID: \"ee1d1f98-eee4-47c2-8ef7-5204fa1d787f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5cee02-7c59cdc9fb-6b47j"
Oct 30 13:24:25.290669 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:25.290639    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee1d1f98-eee4-47c2-8ef7-5204fa1d787f-metrics-certs\") pod \"maas-api-e2e-praxis-a-5cee02-7c59cdc9fb-6b47j\" (UID: \"ee1d1f98-eee4-47c2-8ef7-5204fa1d787f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5cee02-7c59cdc9fb-6b47j"
Oct 30 13:24:25.290669 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:25.290661    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee1d1f98-eee4-47c2-8ef7-5204fa1d787f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-5cee02-7c59cdc9fb-6b47j\" (UID: \"ee1d1f98-eee4-47c2-8ef7-5204fa1d787f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5cee02-7c59cdc9fb-6b47j"
Oct 30 13:24:25.307311 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:25.307277    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnppw\" (UniqueName: \"kubernetes.io/projected/ee1d1f98-eee4-47c2-8ef7-5204fa1d787f-kube-api-access-gnppw\") pod \"maas-api-e2e-praxis-a-5cee02-7c59cdc9fb-6b47j\" (UID: \"ee1d1f98-eee4-47c2-8ef7-5204fa1d787f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5cee02-7c59cdc9fb-6b47j"
Oct 30 13:24:25.348703 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:25.348660    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5cee02-7c59cdc9fb-6b47j"
Oct 30 13:24:25.654604 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:25.654569    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5cee02-7c59cdc9fb-6b47j"]
Oct 30 13:24:25.655469 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:24:25.655438    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee1d1f98_eee4_47c2_8ef7_5204fa1d787f.slice/crio-e553136797760eb6ed398926ca741eb6c165a009f55e2f84c6dbf99557829952 WatchSource:0}: Error finding container e553136797760eb6ed398926ca741eb6c165a009f55e2f84c6dbf99557829952: Status 404 returned error can't find the container with id e553136797760eb6ed398926ca741eb6c165a009f55e2f84c6dbf99557829952
Oct 30 13:24:26.468508 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:26.468478    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e3e617-d9869f767-bmr4t"
Oct 30 13:24:26.500632 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:26.500597    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5cee02-7c59cdc9fb-6b47j" event={"ID":"ee1d1f98-eee4-47c2-8ef7-5204fa1d787f","Type":"ContainerStarted","Data":"8cde7a278617aebee75e41d40ee53029bb92647dcc4d6b262062217b8afaeded"}
Oct 30 13:24:26.500837 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:26.500652    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5cee02-7c59cdc9fb-6b47j" event={"ID":"ee1d1f98-eee4-47c2-8ef7-5204fa1d787f","Type":"ContainerStarted","Data":"e553136797760eb6ed398926ca741eb6c165a009f55e2f84c6dbf99557829952"}
Oct 30 13:24:26.500837 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:26.500675    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5cee02-7c59cdc9fb-6b47j"
Oct 30 13:24:26.555972 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:26.555907    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5cee02-7c59cdc9fb-6b47j" podStartSLOduration=1.242960224 podStartE2EDuration="1.555890199s" totalImagesPullingTime="312.929975ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:24:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:24:26.554571106 +0000 UTC m=+1285.701838348" watchObservedRunningTime="2026-10-30 13:24:26.555890199 +0000 UTC m=+1285.703157441"
Oct 30 13:24:32.516667 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:24:32.516624    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5cee02-7c59cdc9fb-6b47j"
Oct 30 13:24:51.202475 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:24:51.202383    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:25:03.553840 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:03.553803    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4091a5-59f7cb6b7f-7472v"]
Oct 30 13:25:03.559456 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:03.559428    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4091a5-59f7cb6b7f-7472v"
Oct 30 13:25:03.577576 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:03.577543    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4091a5-59f7cb6b7f-7472v"]
Oct 30 13:25:03.584337 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:03.584307    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-4091a5\""
Oct 30 13:25:03.647208 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:03.647173    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30752e68-5a72-423a-8afe-9793414d0c1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-4091a5-59f7cb6b7f-7472v\" (UID: \"30752e68-5a72-423a-8afe-9793414d0c1f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4091a5-59f7cb6b7f-7472v"
Oct 30 13:25:03.647400 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:03.647371    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8dfs\" (UniqueName: \"kubernetes.io/projected/30752e68-5a72-423a-8afe-9793414d0c1f-kube-api-access-t8dfs\") pod \"payload-pre-processing-e2e-praxis-b-4091a5-59f7cb6b7f-7472v\" (UID: \"30752e68-5a72-423a-8afe-9793414d0c1f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4091a5-59f7cb6b7f-7472v"
Oct 30 13:25:03.759154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:03.755043    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8dfs\" (UniqueName: \"kubernetes.io/projected/30752e68-5a72-423a-8afe-9793414d0c1f-kube-api-access-t8dfs\") pod \"payload-pre-processing-e2e-praxis-b-4091a5-59f7cb6b7f-7472v\" (UID: \"30752e68-5a72-423a-8afe-9793414d0c1f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4091a5-59f7cb6b7f-7472v"
Oct 30 13:25:03.759154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:03.755302    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30752e68-5a72-423a-8afe-9793414d0c1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-4091a5-59f7cb6b7f-7472v\" (UID: \"30752e68-5a72-423a-8afe-9793414d0c1f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4091a5-59f7cb6b7f-7472v"
Oct 30 13:25:03.773155 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:03.772837    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30752e68-5a72-423a-8afe-9793414d0c1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-4091a5-59f7cb6b7f-7472v\" (UID: \"30752e68-5a72-423a-8afe-9793414d0c1f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4091a5-59f7cb6b7f-7472v"
Oct 30 13:25:03.813290 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:03.813214    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8dfs\" (UniqueName: \"kubernetes.io/projected/30752e68-5a72-423a-8afe-9793414d0c1f-kube-api-access-t8dfs\") pod \"payload-pre-processing-e2e-praxis-b-4091a5-59f7cb6b7f-7472v\" (UID: \"30752e68-5a72-423a-8afe-9793414d0c1f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4091a5-59f7cb6b7f-7472v"
Oct 30 13:25:03.880920 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:03.879456    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4091a5-59f7cb6b7f-7472v"
Oct 30 13:25:04.212512 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:04.212479    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4091a5-59f7cb6b7f-7472v"]
Oct 30 13:25:04.213152 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:25:04.213109    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30752e68_5a72_423a_8afe_9793414d0c1f.slice/crio-8bdc3ba0682405dc1f1185294932f07fe30437c07e36611123a3b6172b7a8388 WatchSource:0}: Error finding container 8bdc3ba0682405dc1f1185294932f07fe30437c07e36611123a3b6172b7a8388: Status 404 returned error can't find the container with id 8bdc3ba0682405dc1f1185294932f07fe30437c07e36611123a3b6172b7a8388
Oct 30 13:25:04.847916 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:04.847875    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4091a5-59f7cb6b7f-7472v" event={"ID":"30752e68-5a72-423a-8afe-9793414d0c1f","Type":"ContainerStarted","Data":"2aa7e0bf100ba2594769ea6703b2d1fc80307b7b91155c4fdd80f4070059d632"}
Oct 30 13:25:04.848465 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:04.847941    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4091a5-59f7cb6b7f-7472v" event={"ID":"30752e68-5a72-423a-8afe-9793414d0c1f","Type":"ContainerStarted","Data":"8bdc3ba0682405dc1f1185294932f07fe30437c07e36611123a3b6172b7a8388"}
Oct 30 13:25:04.848465 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:04.848081    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4091a5-59f7cb6b7f-7472v"
Oct 30 13:25:04.889852 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:04.889779    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4091a5-59f7cb6b7f-7472v" podStartSLOduration=1.8897584809999999 podStartE2EDuration="1.889758481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:25:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:25:04.887810802 +0000 UTC m=+1324.035078044" watchObservedRunningTime="2026-10-30 13:25:04.889758481 +0000 UTC m=+1324.037025724"
Oct 30 13:25:15.860253 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:15.860217    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4091a5-59f7cb6b7f-7472v"
Oct 30 13:25:21.169286 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:25:21.169244    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:25:25.108182 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:25.108144    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c9e946-8f46bbc55-xmk9s"]
Oct 30 13:25:25.108789 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:25.108423    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9e946-8f46bbc55-xmk9s" podUID="8662720a-f907-42b3-a58e-d37b854a9384" containerName="maas-api" containerID="cri-o://9a1de1abf9d85d930cfd9f61e6c9c6470166f0e8af0b7c57563f432acb820934" gracePeriod=30
Oct 30 13:25:25.508834 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:25.508425    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9e946-8f46bbc55-xmk9s"
Oct 30 13:25:25.580641 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:25.580598    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8662720a-f907-42b3-a58e-d37b854a9384-kube-api-access-pms8r\" (UniqueName: \"kubernetes.io/projected/8662720a-f907-42b3-a58e-d37b854a9384-kube-api-access-pms8r\") pod \"8662720a-f907-42b3-a58e-d37b854a9384\" (UID: \"8662720a-f907-42b3-a58e-d37b854a9384\") "
Oct 30 13:25:25.580962 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:25.580856    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8662720a-f907-42b3-a58e-d37b854a9384-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8662720a-f907-42b3-a58e-d37b854a9384-maas-api-tls\") pod \"8662720a-f907-42b3-a58e-d37b854a9384\" (UID: \"8662720a-f907-42b3-a58e-d37b854a9384\") "
Oct 30 13:25:25.581097 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:25.580962    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8662720a-f907-42b3-a58e-d37b854a9384-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8662720a-f907-42b3-a58e-d37b854a9384-metrics-certs\") pod \"8662720a-f907-42b3-a58e-d37b854a9384\" (UID: \"8662720a-f907-42b3-a58e-d37b854a9384\") "
Oct 30 13:25:25.581097 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:25.581037    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8662720a-f907-42b3-a58e-d37b854a9384-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8662720a-f907-42b3-a58e-d37b854a9384-openshift-service-ca\") pod \"8662720a-f907-42b3-a58e-d37b854a9384\" (UID: \"8662720a-f907-42b3-a58e-d37b854a9384\") "
Oct 30 13:25:25.582355 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:25.582324    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8662720a-f907-42b3-a58e-d37b854a9384-openshift-service-ca" pod "8662720a-f907-42b3-a58e-d37b854a9384" (UID: "8662720a-f907-42b3-a58e-d37b854a9384"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:25:25.617217 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:25.588243    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8662720a-f907-42b3-a58e-d37b854a9384-maas-api-tls" pod "8662720a-f907-42b3-a58e-d37b854a9384" (UID: "8662720a-f907-42b3-a58e-d37b854a9384"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:25:25.628215 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:25.625204    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8662720a-f907-42b3-a58e-d37b854a9384-kube-api-access-pms8r" pod "8662720a-f907-42b3-a58e-d37b854a9384" (UID: "8662720a-f907-42b3-a58e-d37b854a9384"). InnerVolumeSpecName "kube-api-access-pms8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:25.628613 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:25.628552    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8662720a-f907-42b3-a58e-d37b854a9384-metrics-certs" pod "8662720a-f907-42b3-a58e-d37b854a9384" (UID: "8662720a-f907-42b3-a58e-d37b854a9384"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:25:25.682897 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:25.682857    5093 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8662720a-f907-42b3-a58e-d37b854a9384-metrics-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:25:25.682897 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:25.682891    5093 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8662720a-f907-42b3-a58e-d37b854a9384-openshift-service-ca\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:25:25.682897 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:25.682902    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pms8r\" (UniqueName: \"kubernetes.io/projected/8662720a-f907-42b3-a58e-d37b854a9384-kube-api-access-pms8r\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:25:25.683293 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:25.682910    5093 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8662720a-f907-42b3-a58e-d37b854a9384-maas-api-tls\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:25:26.040923 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:26.040883    5093 generic.go:356] "Generic (PLEG): container finished" podID="8662720a-f907-42b3-a58e-d37b854a9384" containerID="9a1de1abf9d85d930cfd9f61e6c9c6470166f0e8af0b7c57563f432acb820934" exitCode=0
Oct 30 13:25:26.040923 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:26.040914    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9e946-8f46bbc55-xmk9s" event={"ID":"8662720a-f907-42b3-a58e-d37b854a9384","Type":"ContainerDied","Data":"9a1de1abf9d85d930cfd9f61e6c9c6470166f0e8af0b7c57563f432acb820934"}
Oct 30 13:25:26.041177 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:26.040980    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9e946-8f46bbc55-xmk9s"
Oct 30 13:25:26.041177 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:26.041002    5093 scope.go:122] "RemoveContainer" containerID="9a1de1abf9d85d930cfd9f61e6c9c6470166f0e8af0b7c57563f432acb820934"
Oct 30 13:25:26.041177 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:26.040990    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9e946-8f46bbc55-xmk9s" event={"ID":"8662720a-f907-42b3-a58e-d37b854a9384","Type":"ContainerDied","Data":"0cb13038ce245f08039edc691504653db141ca0b49451ef599c2b489cfcafa9a"}
Oct 30 13:25:26.053084 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:26.053064    5093 scope.go:122] "RemoveContainer" containerID="9a1de1abf9d85d930cfd9f61e6c9c6470166f0e8af0b7c57563f432acb820934"
Oct 30 13:25:26.053451 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:25:26.053431    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a1de1abf9d85d930cfd9f61e6c9c6470166f0e8af0b7c57563f432acb820934\": container with ID starting with 9a1de1abf9d85d930cfd9f61e6c9c6470166f0e8af0b7c57563f432acb820934 not found: ID does not exist" containerID="9a1de1abf9d85d930cfd9f61e6c9c6470166f0e8af0b7c57563f432acb820934"
Oct 30 13:25:26.053526 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:26.053461    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a1de1abf9d85d930cfd9f61e6c9c6470166f0e8af0b7c57563f432acb820934"} err="failed to get container status \"9a1de1abf9d85d930cfd9f61e6c9c6470166f0e8af0b7c57563f432acb820934\": rpc error: code = NotFound desc = could not find container \"9a1de1abf9d85d930cfd9f61e6c9c6470166f0e8af0b7c57563f432acb820934\": container with ID starting with 9a1de1abf9d85d930cfd9f61e6c9c6470166f0e8af0b7c57563f432acb820934 not found: ID does not exist"
Oct 30 13:25:26.096729 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:26.096697    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c9e946-8f46bbc55-xmk9s"]
Oct 30 13:25:26.110174 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:26.110143    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c9e946-8f46bbc55-xmk9s"]
Oct 30 13:25:27.000836 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:27.000792    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8662720a-f907-42b3-a58e-d37b854a9384" path="/var/lib/kubelet/pods/8662720a-f907-42b3-a58e-d37b854a9384/volumes"
Oct 30 13:25:33.698309 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:33.698273    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw"]
Oct 30 13:25:33.699171 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:33.699119    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw" podUID="20970fe5-df45-4db7-8d63-6c224f8bf7bb" containerName="istio-proxy" containerID="cri-o://b9fd5b49fa6ce35745ac41855ff40d1e608479f3d6bb5e1280dbc9b0eb3e90b4" gracePeriod=30
Oct 30 13:25:37.388847 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:37.388781    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf"]
Oct 30 13:25:37.389373 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:37.389013    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf" podUID="58244ace-ebd4-4b98-929b-8396655992a6" containerName="istio-proxy" containerID="cri-o://49df42e9ffa51680e1f92bf1ba9bdd1bb46d9c0297c19fe0ff7222e2baea779d" gracePeriod=30
Oct 30 13:25:39.007797 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:39.007771    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw"
Oct 30 13:25:39.109982 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:39.109946    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20970fe5-df45-4db7-8d63-6c224f8bf7bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20970fe5-df45-4db7-8d63-6c224f8bf7bb-credential-socket\") pod \"20970fe5-df45-4db7-8d63-6c224f8bf7bb\" (UID: \"20970fe5-df45-4db7-8d63-6c224f8bf7bb\") "
Oct 30 13:25:39.110210 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:39.110096    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20970fe5-df45-4db7-8d63-6c224f8bf7bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20970fe5-df45-4db7-8d63-6c224f8bf7bb-istio-podinfo\") pod \"20970fe5-df45-4db7-8d63-6c224f8bf7bb\" (UID: \"20970fe5-df45-4db7-8d63-6c224f8bf7bb\") "
Oct 30 13:25:39.110210 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:39.110187    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20970fe5-df45-4db7-8d63-6c224f8bf7bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20970fe5-df45-4db7-8d63-6c224f8bf7bb-workload-socket\") pod \"20970fe5-df45-4db7-8d63-6c224f8bf7bb\" (UID: \"20970fe5-df45-4db7-8d63-6c224f8bf7bb\") "
Oct 30 13:25:39.110339 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:39.110249    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20970fe5-df45-4db7-8d63-6c224f8bf7bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20970fe5-df45-4db7-8d63-6c224f8bf7bb-istio-data\") pod \"20970fe5-df45-4db7-8d63-6c224f8bf7bb\" (UID: \"20970fe5-df45-4db7-8d63-6c224f8bf7bb\") "
Oct 30 13:25:39.110339 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:39.110294    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20970fe5-df45-4db7-8d63-6c224f8bf7bb-kube-api-access-sr5fc\" (UniqueName: \"kubernetes.io/projected/20970fe5-df45-4db7-8d63-6c224f8bf7bb-kube-api-access-sr5fc\") pod \"20970fe5-df45-4db7-8d63-6c224f8bf7bb\" (UID: \"20970fe5-df45-4db7-8d63-6c224f8bf7bb\") "
Oct 30 13:25:39.110339 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:39.110309    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20970fe5-df45-4db7-8d63-6c224f8bf7bb-credential-socket" pod "20970fe5-df45-4db7-8d63-6c224f8bf7bb" (UID: "20970fe5-df45-4db7-8d63-6c224f8bf7bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:39.110999 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:39.110489    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20970fe5-df45-4db7-8d63-6c224f8bf7bb-istio-token\" (UniqueName: \"kubernetes.io/projected/20970fe5-df45-4db7-8d63-6c224f8bf7bb-istio-token\") pod \"20970fe5-df45-4db7-8d63-6c224f8bf7bb\" (UID: \"20970fe5-df45-4db7-8d63-6c224f8bf7bb\") "
Oct 30 13:25:39.110999 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:39.110536    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20970fe5-df45-4db7-8d63-6c224f8bf7bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20970fe5-df45-4db7-8d63-6c224f8bf7bb-workload-certs\") pod \"20970fe5-df45-4db7-8d63-6c224f8bf7bb\" (UID: \"20970fe5-df45-4db7-8d63-6c224f8bf7bb\") "
Oct 30 13:25:39.110999 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:39.110579    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20970fe5-df45-4db7-8d63-6c224f8bf7bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20970fe5-df45-4db7-8d63-6c224f8bf7bb-istio-envoy\") pod \"20970fe5-df45-4db7-8d63-6c224f8bf7bb\" (UID: \"20970fe5-df45-4db7-8d63-6c224f8bf7bb\") "
Oct 30 13:25:39.110999 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:39.110618    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20970fe5-df45-4db7-8d63-6c224f8bf7bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20970fe5-df45-4db7-8d63-6c224f8bf7bb-istiod-ca-cert\") pod \"20970fe5-df45-4db7-8d63-6c224f8bf7bb\" (UID: \"20970fe5-df45-4db7-8d63-6c224f8bf7bb\") "
Oct 30 13:25:39.110999 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:39.110703    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20970fe5-df45-4db7-8d63-6c224f8bf7bb-istio-data" pod "20970fe5-df45-4db7-8d63-6c224f8bf7bb" (UID: "20970fe5-df45-4db7-8d63-6c224f8bf7bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:39.110999 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:39.110867    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20970fe5-df45-4db7-8d63-6c224f8bf7bb-workload-certs" pod "20970fe5-df45-4db7-8d63-6c224f8bf7bb" (UID: "20970fe5-df45-4db7-8d63-6c224f8bf7bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:39.111586 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:39.111243    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20970fe5-df45-4db7-8d63-6c224f8bf7bb-istiod-ca-cert" pod "20970fe5-df45-4db7-8d63-6c224f8bf7bb" (UID: "20970fe5-df45-4db7-8d63-6c224f8bf7bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:25:39.111586 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:39.111500    5093 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20970fe5-df45-4db7-8d63-6c224f8bf7bb-workload-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:25:39.111586 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:39.111525    5093 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20970fe5-df45-4db7-8d63-6c224f8bf7bb-istiod-ca-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:25:39.111586 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:39.111539    5093 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20970fe5-df45-4db7-8d63-6c224f8bf7bb-credential-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:25:39.111586 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:39.111553    5093 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20970fe5-df45-4db7-8d63-6c224f8bf7bb-istio-data\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:25:39.111865 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:39.111629    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20970fe5-df45-4db7-8d63-6c224f8bf7bb-workload-socket" pod "20970fe5-df45-4db7-8d63-6c224f8bf7bb" (UID: "20970fe5-df45-4db7-8d63-6c224f8bf7bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:39.113645 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:39.113606    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20970fe5-df45-4db7-8d63-6c224f8bf7bb-istio-envoy" pod "20970fe5-df45-4db7-8d63-6c224f8bf7bb" (UID: "20970fe5-df45-4db7-8d63-6c224f8bf7bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:39.113645 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:39.113635    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20970fe5-df45-4db7-8d63-6c224f8bf7bb-istio-podinfo" pod "20970fe5-df45-4db7-8d63-6c224f8bf7bb" (UID: "20970fe5-df45-4db7-8d63-6c224f8bf7bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:25:39.114892 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:39.114863    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20970fe5-df45-4db7-8d63-6c224f8bf7bb-kube-api-access-sr5fc" pod "20970fe5-df45-4db7-8d63-6c224f8bf7bb" (UID: "20970fe5-df45-4db7-8d63-6c224f8bf7bb"). InnerVolumeSpecName "kube-api-access-sr5fc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:39.115475 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:39.115451    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20970fe5-df45-4db7-8d63-6c224f8bf7bb-istio-token" pod "20970fe5-df45-4db7-8d63-6c224f8bf7bb" (UID: "20970fe5-df45-4db7-8d63-6c224f8bf7bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:39.151534 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:39.151498    5093 generic.go:356] "Generic (PLEG): container finished" podID="20970fe5-df45-4db7-8d63-6c224f8bf7bb" containerID="b9fd5b49fa6ce35745ac41855ff40d1e608479f3d6bb5e1280dbc9b0eb3e90b4" exitCode=0
Oct 30 13:25:39.151703 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:39.151558    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw"
Oct 30 13:25:39.151703 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:39.151563    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw" event={"ID":"20970fe5-df45-4db7-8d63-6c224f8bf7bb","Type":"ContainerDied","Data":"b9fd5b49fa6ce35745ac41855ff40d1e608479f3d6bb5e1280dbc9b0eb3e90b4"}
Oct 30 13:25:39.151703 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:39.151623    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw" event={"ID":"20970fe5-df45-4db7-8d63-6c224f8bf7bb","Type":"ContainerDied","Data":"8aaf968a5148c0da0f55a7efdbe086278e9906ceed6474b8c1068382443be1cb"}
Oct 30 13:25:39.151703 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:39.151653    5093 scope.go:122] "RemoveContainer" containerID="b9fd5b49fa6ce35745ac41855ff40d1e608479f3d6bb5e1280dbc9b0eb3e90b4"
Oct 30 13:25:39.163345 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:39.163322    5093 scope.go:122] "RemoveContainer" containerID="c1c6ab2defcaedba8959f85583421e3c8c6de7d18786ba04c9a02f54c7de30ed"
Oct 30 13:25:39.174206 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:39.174186    5093 scope.go:122] "RemoveContainer" containerID="b9fd5b49fa6ce35745ac41855ff40d1e608479f3d6bb5e1280dbc9b0eb3e90b4"
Oct 30 13:25:39.174498 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:25:39.174479    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9fd5b49fa6ce35745ac41855ff40d1e608479f3d6bb5e1280dbc9b0eb3e90b4\": container with ID starting with b9fd5b49fa6ce35745ac41855ff40d1e608479f3d6bb5e1280dbc9b0eb3e90b4 not found: ID does not exist" containerID="b9fd5b49fa6ce35745ac41855ff40d1e608479f3d6bb5e1280dbc9b0eb3e90b4"
Oct 30 13:25:39.174589 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:39.174512    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9fd5b49fa6ce35745ac41855ff40d1e608479f3d6bb5e1280dbc9b0eb3e90b4"} err="failed to get container status \"b9fd5b49fa6ce35745ac41855ff40d1e608479f3d6bb5e1280dbc9b0eb3e90b4\": rpc error: code = NotFound desc = could not find container \"b9fd5b49fa6ce35745ac41855ff40d1e608479f3d6bb5e1280dbc9b0eb3e90b4\": container with ID starting with b9fd5b49fa6ce35745ac41855ff40d1e608479f3d6bb5e1280dbc9b0eb3e90b4 not found: ID does not exist"
Oct 30 13:25:39.174589 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:39.174537    5093 scope.go:122] "RemoveContainer" containerID="c1c6ab2defcaedba8959f85583421e3c8c6de7d18786ba04c9a02f54c7de30ed"
Oct 30 13:25:39.174790 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:25:39.174774    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1c6ab2defcaedba8959f85583421e3c8c6de7d18786ba04c9a02f54c7de30ed\": container with ID starting with c1c6ab2defcaedba8959f85583421e3c8c6de7d18786ba04c9a02f54c7de30ed not found: ID does not exist" containerID="c1c6ab2defcaedba8959f85583421e3c8c6de7d18786ba04c9a02f54c7de30ed"
Oct 30 13:25:39.174845 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:39.174799    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1c6ab2defcaedba8959f85583421e3c8c6de7d18786ba04c9a02f54c7de30ed"} err="failed to get container status \"c1c6ab2defcaedba8959f85583421e3c8c6de7d18786ba04c9a02f54c7de30ed\": rpc error: code = NotFound desc = could not find container \"c1c6ab2defcaedba8959f85583421e3c8c6de7d18786ba04c9a02f54c7de30ed\": container with ID starting with c1c6ab2defcaedba8959f85583421e3c8c6de7d18786ba04c9a02f54c7de30ed not found: ID does not exist"
Oct 30 13:25:39.209328 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:39.209297    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw"]
Oct 30 13:25:39.212432 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:39.212405    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20970fe5-df45-4db7-8d63-6c224f8bf7bb-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:25:39.212432 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:39.212431    5093 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20970fe5-df45-4db7-8d63-6c224f8bf7bb-istio-envoy\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:25:39.212432 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:39.212440    5093 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20970fe5-df45-4db7-8d63-6c224f8bf7bb-istio-podinfo\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:25:39.212624 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:39.212451    5093 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20970fe5-df45-4db7-8d63-6c224f8bf7bb-workload-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:25:39.212624 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:39.212459    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sr5fc\" (UniqueName: \"kubernetes.io/projected/20970fe5-df45-4db7-8d63-6c224f8bf7bb-kube-api-access-sr5fc\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:25:39.214540 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:39.214519    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c9e946-openshift-default-66f5d8cb57-259dw"]
Oct 30 13:25:40.999618 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:40.999583    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20970fe5-df45-4db7-8d63-6c224f8bf7bb" path="/var/lib/kubelet/pods/20970fe5-df45-4db7-8d63-6c224f8bf7bb/volumes"
Oct 30 13:25:42.697773 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:42.697744    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf"
Oct 30 13:25:42.772004 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:42.771968    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58244ace-ebd4-4b98-929b-8396655992a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58244ace-ebd4-4b98-929b-8396655992a6-workload-certs\") pod \"58244ace-ebd4-4b98-929b-8396655992a6\" (UID: \"58244ace-ebd4-4b98-929b-8396655992a6\") "
Oct 30 13:25:42.772300 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:42.772278    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58244ace-ebd4-4b98-929b-8396655992a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58244ace-ebd4-4b98-929b-8396655992a6-istio-podinfo\") pod \"58244ace-ebd4-4b98-929b-8396655992a6\" (UID: \"58244ace-ebd4-4b98-929b-8396655992a6\") "
Oct 30 13:25:42.772477 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:42.772460    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58244ace-ebd4-4b98-929b-8396655992a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58244ace-ebd4-4b98-929b-8396655992a6-istio-data\") pod \"58244ace-ebd4-4b98-929b-8396655992a6\" (UID: \"58244ace-ebd4-4b98-929b-8396655992a6\") "
Oct 30 13:25:42.772707 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:42.772675    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58244ace-ebd4-4b98-929b-8396655992a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58244ace-ebd4-4b98-929b-8396655992a6-credential-socket\") pod \"58244ace-ebd4-4b98-929b-8396655992a6\" (UID: \"58244ace-ebd4-4b98-929b-8396655992a6\") "
Oct 30 13:25:42.772801 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:42.772725    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58244ace-ebd4-4b98-929b-8396655992a6-kube-api-access-v42gh\" (UniqueName: \"kubernetes.io/projected/58244ace-ebd4-4b98-929b-8396655992a6-kube-api-access-v42gh\") pod \"58244ace-ebd4-4b98-929b-8396655992a6\" (UID: \"58244ace-ebd4-4b98-929b-8396655992a6\") "
Oct 30 13:25:42.772801 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:42.772761    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58244ace-ebd4-4b98-929b-8396655992a6-istio-token\" (UniqueName: \"kubernetes.io/projected/58244ace-ebd4-4b98-929b-8396655992a6-istio-token\") pod \"58244ace-ebd4-4b98-929b-8396655992a6\" (UID: \"58244ace-ebd4-4b98-929b-8396655992a6\") "
Oct 30 13:25:42.772905 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:42.772840    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58244ace-ebd4-4b98-929b-8396655992a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58244ace-ebd4-4b98-929b-8396655992a6-istio-envoy\") pod \"58244ace-ebd4-4b98-929b-8396655992a6\" (UID: \"58244ace-ebd4-4b98-929b-8396655992a6\") "
Oct 30 13:25:42.772972 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:42.772938    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58244ace-ebd4-4b98-929b-8396655992a6-istio-data" pod "58244ace-ebd4-4b98-929b-8396655992a6" (UID: "58244ace-ebd4-4b98-929b-8396655992a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:42.773053 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:42.772948    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58244ace-ebd4-4b98-929b-8396655992a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58244ace-ebd4-4b98-929b-8396655992a6-workload-socket\") pod \"58244ace-ebd4-4b98-929b-8396655992a6\" (UID: \"58244ace-ebd4-4b98-929b-8396655992a6\") "
Oct 30 13:25:42.773116 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:42.773062    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58244ace-ebd4-4b98-929b-8396655992a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58244ace-ebd4-4b98-929b-8396655992a6-istiod-ca-cert\") pod \"58244ace-ebd4-4b98-929b-8396655992a6\" (UID: \"58244ace-ebd4-4b98-929b-8396655992a6\") "
Oct 30 13:25:42.773224 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:42.773196    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58244ace-ebd4-4b98-929b-8396655992a6-workload-socket" pod "58244ace-ebd4-4b98-929b-8396655992a6" (UID: "58244ace-ebd4-4b98-929b-8396655992a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:42.773416 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:42.773395    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58244ace-ebd4-4b98-929b-8396655992a6-credential-socket" pod "58244ace-ebd4-4b98-929b-8396655992a6" (UID: "58244ace-ebd4-4b98-929b-8396655992a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:42.774012 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:42.773725    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58244ace-ebd4-4b98-929b-8396655992a6-istiod-ca-cert" pod "58244ace-ebd4-4b98-929b-8396655992a6" (UID: "58244ace-ebd4-4b98-929b-8396655992a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:25:42.774012 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:42.773784    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58244ace-ebd4-4b98-929b-8396655992a6-workload-certs" pod "58244ace-ebd4-4b98-929b-8396655992a6" (UID: "58244ace-ebd4-4b98-929b-8396655992a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:42.774012 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:42.773819    5093 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58244ace-ebd4-4b98-929b-8396655992a6-workload-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:25:42.774012 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:42.773858    5093 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58244ace-ebd4-4b98-929b-8396655992a6-istio-data\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:25:42.774012 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:42.773873    5093 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58244ace-ebd4-4b98-929b-8396655992a6-credential-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:25:42.776535 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:42.776504    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58244ace-ebd4-4b98-929b-8396655992a6-kube-api-access-v42gh" pod "58244ace-ebd4-4b98-929b-8396655992a6" (UID: "58244ace-ebd4-4b98-929b-8396655992a6"). InnerVolumeSpecName "kube-api-access-v42gh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:42.776645 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:42.776588    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58244ace-ebd4-4b98-929b-8396655992a6-istio-envoy" pod "58244ace-ebd4-4b98-929b-8396655992a6" (UID: "58244ace-ebd4-4b98-929b-8396655992a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:42.776771 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:42.776754    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58244ace-ebd4-4b98-929b-8396655992a6-istio-token" pod "58244ace-ebd4-4b98-929b-8396655992a6" (UID: "58244ace-ebd4-4b98-929b-8396655992a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:42.777749 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:42.777722    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58244ace-ebd4-4b98-929b-8396655992a6-istio-podinfo" pod "58244ace-ebd4-4b98-929b-8396655992a6" (UID: "58244ace-ebd4-4b98-929b-8396655992a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:25:42.874851 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:42.874812    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v42gh\" (UniqueName: \"kubernetes.io/projected/58244ace-ebd4-4b98-929b-8396655992a6-kube-api-access-v42gh\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:25:42.874851 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:42.874847    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58244ace-ebd4-4b98-929b-8396655992a6-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:25:42.874851 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:42.874859    5093 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58244ace-ebd4-4b98-929b-8396655992a6-istio-envoy\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:25:42.875117 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:42.874866    5093 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58244ace-ebd4-4b98-929b-8396655992a6-istiod-ca-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:25:42.875117 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:42.874875    5093 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58244ace-ebd4-4b98-929b-8396655992a6-workload-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:25:42.875117 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:42.874882    5093 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58244ace-ebd4-4b98-929b-8396655992a6-istio-podinfo\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:25:43.186270 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:43.186230    5093 generic.go:356] "Generic (PLEG): container finished" podID="58244ace-ebd4-4b98-929b-8396655992a6" containerID="49df42e9ffa51680e1f92bf1ba9bdd1bb46d9c0297c19fe0ff7222e2baea779d" exitCode=0
Oct 30 13:25:43.186443 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:43.186283    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf"
Oct 30 13:25:43.186443 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:43.186315    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf" event={"ID":"58244ace-ebd4-4b98-929b-8396655992a6","Type":"ContainerDied","Data":"49df42e9ffa51680e1f92bf1ba9bdd1bb46d9c0297c19fe0ff7222e2baea779d"}
Oct 30 13:25:43.186443 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:43.186371    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf" event={"ID":"58244ace-ebd4-4b98-929b-8396655992a6","Type":"ContainerDied","Data":"dfcec110b9514bafbd5c1ae12ca3984c12b1aad891e1db8b78b0d6b64fb01c6e"}
Oct 30 13:25:43.186443 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:43.186393    5093 scope.go:122] "RemoveContainer" containerID="49df42e9ffa51680e1f92bf1ba9bdd1bb46d9c0297c19fe0ff7222e2baea779d"
Oct 30 13:25:43.196639 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:43.196620    5093 scope.go:122] "RemoveContainer" containerID="bb40ce33d522380835ac6b8cf3867afdbf64bdf3052b03b320cfa389404fdf81"
Oct 30 13:25:43.206012 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:43.205984    5093 scope.go:122] "RemoveContainer" containerID="49df42e9ffa51680e1f92bf1ba9bdd1bb46d9c0297c19fe0ff7222e2baea779d"
Oct 30 13:25:43.206368 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:25:43.206347    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49df42e9ffa51680e1f92bf1ba9bdd1bb46d9c0297c19fe0ff7222e2baea779d\": container with ID starting with 49df42e9ffa51680e1f92bf1ba9bdd1bb46d9c0297c19fe0ff7222e2baea779d not found: ID does not exist" containerID="49df42e9ffa51680e1f92bf1ba9bdd1bb46d9c0297c19fe0ff7222e2baea779d"
Oct 30 13:25:43.206418 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:43.206381    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49df42e9ffa51680e1f92bf1ba9bdd1bb46d9c0297c19fe0ff7222e2baea779d"} err="failed to get container status \"49df42e9ffa51680e1f92bf1ba9bdd1bb46d9c0297c19fe0ff7222e2baea779d\": rpc error: code = NotFound desc = could not find container \"49df42e9ffa51680e1f92bf1ba9bdd1bb46d9c0297c19fe0ff7222e2baea779d\": container with ID starting with 49df42e9ffa51680e1f92bf1ba9bdd1bb46d9c0297c19fe0ff7222e2baea779d not found: ID does not exist"
Oct 30 13:25:43.206418 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:43.206401    5093 scope.go:122] "RemoveContainer" containerID="bb40ce33d522380835ac6b8cf3867afdbf64bdf3052b03b320cfa389404fdf81"
Oct 30 13:25:43.206656 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:25:43.206641    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb40ce33d522380835ac6b8cf3867afdbf64bdf3052b03b320cfa389404fdf81\": container with ID starting with bb40ce33d522380835ac6b8cf3867afdbf64bdf3052b03b320cfa389404fdf81 not found: ID does not exist" containerID="bb40ce33d522380835ac6b8cf3867afdbf64bdf3052b03b320cfa389404fdf81"
Oct 30 13:25:43.206695 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:43.206660    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb40ce33d522380835ac6b8cf3867afdbf64bdf3052b03b320cfa389404fdf81"} err="failed to get container status \"bb40ce33d522380835ac6b8cf3867afdbf64bdf3052b03b320cfa389404fdf81\": rpc error: code = NotFound desc = could not find container \"bb40ce33d522380835ac6b8cf3867afdbf64bdf3052b03b320cfa389404fdf81\": container with ID starting with bb40ce33d522380835ac6b8cf3867afdbf64bdf3052b03b320cfa389404fdf81 not found: ID does not exist"
Oct 30 13:25:43.255234 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:43.255198    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf"]
Oct 30 13:25:43.268706 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:43.268634    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-3a82db-openshift-default-5c89984dc7-8n5sf"]
Oct 30 13:25:45.004220 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:45.004178    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58244ace-ebd4-4b98-929b-8396655992a6" path="/var/lib/kubelet/pods/58244ace-ebd4-4b98-929b-8396655992a6/volumes"
Oct 30 13:25:51.167317 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:25:51.167272    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:25:52.338178 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:52.338106    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95"]
Oct 30 13:25:52.338823 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:52.338416    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95" podUID="ca419c67-141b-4a04-8ee6-95301302c378" containerName="istio-proxy" containerID="cri-o://d352e3cc94776beea1f3b487699b559518c2f961a9e689a652349c138081e793" gracePeriod=30
Oct 30 13:25:55.665430 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:55.665390    5093 patch_prober.go:29] interesting pod/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:25:55.665855 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:55.665464    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95" podUID="ca419c67-141b-4a04-8ee6-95301302c378" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:25:57.646248 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:57.646218    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95"
Oct 30 13:25:57.726409 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:57.726372    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ca419c67-141b-4a04-8ee6-95301302c378-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca419c67-141b-4a04-8ee6-95301302c378-istio-podinfo\") pod \"ca419c67-141b-4a04-8ee6-95301302c378\" (UID: \"ca419c67-141b-4a04-8ee6-95301302c378\") "
Oct 30 13:25:57.726609 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:57.726434    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca419c67-141b-4a04-8ee6-95301302c378-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca419c67-141b-4a04-8ee6-95301302c378-istio-envoy\") pod \"ca419c67-141b-4a04-8ee6-95301302c378\" (UID: \"ca419c67-141b-4a04-8ee6-95301302c378\") "
Oct 30 13:25:57.726609 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:57.726500    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca419c67-141b-4a04-8ee6-95301302c378-kube-api-access-c5xk4\" (UniqueName: \"kubernetes.io/projected/ca419c67-141b-4a04-8ee6-95301302c378-kube-api-access-c5xk4\") pod \"ca419c67-141b-4a04-8ee6-95301302c378\" (UID: \"ca419c67-141b-4a04-8ee6-95301302c378\") "
Oct 30 13:25:57.726609 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:57.726547    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca419c67-141b-4a04-8ee6-95301302c378-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca419c67-141b-4a04-8ee6-95301302c378-istiod-ca-cert\") pod \"ca419c67-141b-4a04-8ee6-95301302c378\" (UID: \"ca419c67-141b-4a04-8ee6-95301302c378\") "
Oct 30 13:25:57.726811 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:57.726637    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca419c67-141b-4a04-8ee6-95301302c378-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca419c67-141b-4a04-8ee6-95301302c378-workload-certs\") pod \"ca419c67-141b-4a04-8ee6-95301302c378\" (UID: \"ca419c67-141b-4a04-8ee6-95301302c378\") "
Oct 30 13:25:57.726811 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:57.726692    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca419c67-141b-4a04-8ee6-95301302c378-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca419c67-141b-4a04-8ee6-95301302c378-istio-data\") pod \"ca419c67-141b-4a04-8ee6-95301302c378\" (UID: \"ca419c67-141b-4a04-8ee6-95301302c378\") "
Oct 30 13:25:57.726811 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:57.726730    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca419c67-141b-4a04-8ee6-95301302c378-istio-token\" (UniqueName: \"kubernetes.io/projected/ca419c67-141b-4a04-8ee6-95301302c378-istio-token\") pod \"ca419c67-141b-4a04-8ee6-95301302c378\" (UID: \"ca419c67-141b-4a04-8ee6-95301302c378\") "
Oct 30 13:25:57.726980 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:57.726812    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca419c67-141b-4a04-8ee6-95301302c378-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca419c67-141b-4a04-8ee6-95301302c378-workload-socket\") pod \"ca419c67-141b-4a04-8ee6-95301302c378\" (UID: \"ca419c67-141b-4a04-8ee6-95301302c378\") "
Oct 30 13:25:57.726980 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:57.726874    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca419c67-141b-4a04-8ee6-95301302c378-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca419c67-141b-4a04-8ee6-95301302c378-credential-socket\") pod \"ca419c67-141b-4a04-8ee6-95301302c378\" (UID: \"ca419c67-141b-4a04-8ee6-95301302c378\") "
Oct 30 13:25:57.727695 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:57.727668    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca419c67-141b-4a04-8ee6-95301302c378-credential-socket" pod "ca419c67-141b-4a04-8ee6-95301302c378" (UID: "ca419c67-141b-4a04-8ee6-95301302c378"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:57.727888 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:57.727855    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca419c67-141b-4a04-8ee6-95301302c378-istio-data" pod "ca419c67-141b-4a04-8ee6-95301302c378" (UID: "ca419c67-141b-4a04-8ee6-95301302c378"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:57.728191 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:57.728167    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca419c67-141b-4a04-8ee6-95301302c378-workload-certs" pod "ca419c67-141b-4a04-8ee6-95301302c378" (UID: "ca419c67-141b-4a04-8ee6-95301302c378"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:57.728294 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:57.728267    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca419c67-141b-4a04-8ee6-95301302c378-istiod-ca-cert" pod "ca419c67-141b-4a04-8ee6-95301302c378" (UID: "ca419c67-141b-4a04-8ee6-95301302c378"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:25:57.728385 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:57.728361    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca419c67-141b-4a04-8ee6-95301302c378-workload-socket" pod "ca419c67-141b-4a04-8ee6-95301302c378" (UID: "ca419c67-141b-4a04-8ee6-95301302c378"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:57.730835 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:57.730798    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca419c67-141b-4a04-8ee6-95301302c378-istio-podinfo" pod "ca419c67-141b-4a04-8ee6-95301302c378" (UID: "ca419c67-141b-4a04-8ee6-95301302c378"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:25:57.731609 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:57.731577    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca419c67-141b-4a04-8ee6-95301302c378-kube-api-access-c5xk4" pod "ca419c67-141b-4a04-8ee6-95301302c378" (UID: "ca419c67-141b-4a04-8ee6-95301302c378"). InnerVolumeSpecName "kube-api-access-c5xk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:57.732163 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:57.732118    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca419c67-141b-4a04-8ee6-95301302c378-istio-token" pod "ca419c67-141b-4a04-8ee6-95301302c378" (UID: "ca419c67-141b-4a04-8ee6-95301302c378"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:57.732563 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:57.732540    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca419c67-141b-4a04-8ee6-95301302c378-istio-envoy" pod "ca419c67-141b-4a04-8ee6-95301302c378" (UID: "ca419c67-141b-4a04-8ee6-95301302c378"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:57.831349 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:57.831309    5093 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca419c67-141b-4a04-8ee6-95301302c378-workload-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:25:57.831349 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:57.831346    5093 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca419c67-141b-4a04-8ee6-95301302c378-credential-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:25:57.831349 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:57.831361    5093 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca419c67-141b-4a04-8ee6-95301302c378-istio-podinfo\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:25:57.831770 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:57.831374    5093 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca419c67-141b-4a04-8ee6-95301302c378-istio-envoy\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:25:57.831770 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:57.831387    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5xk4\" (UniqueName: \"kubernetes.io/projected/ca419c67-141b-4a04-8ee6-95301302c378-kube-api-access-c5xk4\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:25:57.831770 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:57.831401    5093 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca419c67-141b-4a04-8ee6-95301302c378-istiod-ca-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:25:57.831770 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:57.831414    5093 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca419c67-141b-4a04-8ee6-95301302c378-workload-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:25:57.831770 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:57.831425    5093 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca419c67-141b-4a04-8ee6-95301302c378-istio-data\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:25:57.831770 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:57.831437    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca419c67-141b-4a04-8ee6-95301302c378-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:25:58.332218 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:58.332157    5093 generic.go:356] "Generic (PLEG): container finished" podID="ca419c67-141b-4a04-8ee6-95301302c378" containerID="d352e3cc94776beea1f3b487699b559518c2f961a9e689a652349c138081e793" exitCode=0
Oct 30 13:25:58.332218 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:58.332208    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95"
Oct 30 13:25:58.332218 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:58.332205    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95" event={"ID":"ca419c67-141b-4a04-8ee6-95301302c378","Type":"ContainerDied","Data":"d352e3cc94776beea1f3b487699b559518c2f961a9e689a652349c138081e793"}
Oct 30 13:25:58.332527 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:58.332282    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95" event={"ID":"ca419c67-141b-4a04-8ee6-95301302c378","Type":"ContainerDied","Data":"553d73e3eb865d3a4620bdf58b59f4a09a89cfc058b57c39c823ddc0b42d859b"}
Oct 30 13:25:58.332527 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:58.332315    5093 scope.go:122] "RemoveContainer" containerID="d352e3cc94776beea1f3b487699b559518c2f961a9e689a652349c138081e793"
Oct 30 13:25:58.345519 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:58.345500    5093 scope.go:122] "RemoveContainer" containerID="5b70a8c379098929a76f02957700146aa58a065c0ca6ce32a4f4e7d72c53427e"
Oct 30 13:25:58.355584 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:58.355566    5093 scope.go:122] "RemoveContainer" containerID="d352e3cc94776beea1f3b487699b559518c2f961a9e689a652349c138081e793"
Oct 30 13:25:58.355905 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:25:58.355881    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d352e3cc94776beea1f3b487699b559518c2f961a9e689a652349c138081e793\": container with ID starting with d352e3cc94776beea1f3b487699b559518c2f961a9e689a652349c138081e793 not found: ID does not exist" containerID="d352e3cc94776beea1f3b487699b559518c2f961a9e689a652349c138081e793"
Oct 30 13:25:58.355970 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:58.355921    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d352e3cc94776beea1f3b487699b559518c2f961a9e689a652349c138081e793"} err="failed to get container status \"d352e3cc94776beea1f3b487699b559518c2f961a9e689a652349c138081e793\": rpc error: code = NotFound desc = could not find container \"d352e3cc94776beea1f3b487699b559518c2f961a9e689a652349c138081e793\": container with ID starting with d352e3cc94776beea1f3b487699b559518c2f961a9e689a652349c138081e793 not found: ID does not exist"
Oct 30 13:25:58.355970 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:58.355945    5093 scope.go:122] "RemoveContainer" containerID="5b70a8c379098929a76f02957700146aa58a065c0ca6ce32a4f4e7d72c53427e"
Oct 30 13:25:58.356371 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:25:58.356351    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b70a8c379098929a76f02957700146aa58a065c0ca6ce32a4f4e7d72c53427e\": container with ID starting with 5b70a8c379098929a76f02957700146aa58a065c0ca6ce32a4f4e7d72c53427e not found: ID does not exist" containerID="5b70a8c379098929a76f02957700146aa58a065c0ca6ce32a4f4e7d72c53427e"
Oct 30 13:25:58.356442 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:58.356381    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b70a8c379098929a76f02957700146aa58a065c0ca6ce32a4f4e7d72c53427e"} err="failed to get container status \"5b70a8c379098929a76f02957700146aa58a065c0ca6ce32a4f4e7d72c53427e\": rpc error: code = NotFound desc = could not find container \"5b70a8c379098929a76f02957700146aa58a065c0ca6ce32a4f4e7d72c53427e\": container with ID starting with 5b70a8c379098929a76f02957700146aa58a065c0ca6ce32a4f4e7d72c53427e not found: ID does not exist"
Oct 30 13:25:58.387360 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:58.387330    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95"]
Oct 30 13:25:58.400472 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:58.400445    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-8fb1a8-openshift-default-67b547b679-t6d95"]
Oct 30 13:25:59.001385 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:25:59.001354    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca419c67-141b-4a04-8ee6-95301302c378" path="/var/lib/kubelet/pods/ca419c67-141b-4a04-8ee6-95301302c378/volumes"
Oct 30 13:26:13.212895 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:13.212803    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e3e617-d9869f767-bmr4t"]
Oct 30 13:26:13.213397 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:13.213064    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e3e617-d9869f767-bmr4t" podUID="aadc22cb-ee34-43f9-9071-2d9f86044235" containerName="maas-api" containerID="cri-o://afd87389dce0fea565a85a307fa2f03e83d13f15992491ba8192af962aba18db" gracePeriod=30
Oct 30 13:26:13.584328 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:13.584241    5093 generic.go:356] "Generic (PLEG): container finished" podID="aadc22cb-ee34-43f9-9071-2d9f86044235" containerID="afd87389dce0fea565a85a307fa2f03e83d13f15992491ba8192af962aba18db" exitCode=0
Oct 30 13:26:13.584328 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:13.584267    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e3e617-d9869f767-bmr4t" event={"ID":"aadc22cb-ee34-43f9-9071-2d9f86044235","Type":"ContainerDied","Data":"afd87389dce0fea565a85a307fa2f03e83d13f15992491ba8192af962aba18db"}
Oct 30 13:26:13.689667 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:13.689631    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e3e617-d9869f767-bmr4t"
Oct 30 13:26:13.805186 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:13.805116    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aadc22cb-ee34-43f9-9071-2d9f86044235-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aadc22cb-ee34-43f9-9071-2d9f86044235-openshift-service-ca\") pod \"aadc22cb-ee34-43f9-9071-2d9f86044235\" (UID: \"aadc22cb-ee34-43f9-9071-2d9f86044235\") "
Oct 30 13:26:13.805380 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:13.805280    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aadc22cb-ee34-43f9-9071-2d9f86044235-kube-api-access-qwkdv\" (UniqueName: \"kubernetes.io/projected/aadc22cb-ee34-43f9-9071-2d9f86044235-kube-api-access-qwkdv\") pod \"aadc22cb-ee34-43f9-9071-2d9f86044235\" (UID: \"aadc22cb-ee34-43f9-9071-2d9f86044235\") "
Oct 30 13:26:13.805380 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:13.805369    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aadc22cb-ee34-43f9-9071-2d9f86044235-metrics-certs\" (UniqueName: \"kubernetes.io/secret/aadc22cb-ee34-43f9-9071-2d9f86044235-metrics-certs\") pod \"aadc22cb-ee34-43f9-9071-2d9f86044235\" (UID: \"aadc22cb-ee34-43f9-9071-2d9f86044235\") "
Oct 30 13:26:13.805506 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:13.805395    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aadc22cb-ee34-43f9-9071-2d9f86044235-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aadc22cb-ee34-43f9-9071-2d9f86044235-maas-api-tls\") pod \"aadc22cb-ee34-43f9-9071-2d9f86044235\" (UID: \"aadc22cb-ee34-43f9-9071-2d9f86044235\") "
Oct 30 13:26:13.805506 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:13.805438    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aadc22cb-ee34-43f9-9071-2d9f86044235-openshift-service-ca" pod "aadc22cb-ee34-43f9-9071-2d9f86044235" (UID: "aadc22cb-ee34-43f9-9071-2d9f86044235"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:26:13.805862 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:13.805830    5093 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aadc22cb-ee34-43f9-9071-2d9f86044235-openshift-service-ca\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:26:13.807634 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:13.807605    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aadc22cb-ee34-43f9-9071-2d9f86044235-metrics-certs" pod "aadc22cb-ee34-43f9-9071-2d9f86044235" (UID: "aadc22cb-ee34-43f9-9071-2d9f86044235"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:26:13.807741 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:13.807649    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aadc22cb-ee34-43f9-9071-2d9f86044235-maas-api-tls" pod "aadc22cb-ee34-43f9-9071-2d9f86044235" (UID: "aadc22cb-ee34-43f9-9071-2d9f86044235"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:26:13.807741 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:13.807707    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aadc22cb-ee34-43f9-9071-2d9f86044235-kube-api-access-qwkdv" pod "aadc22cb-ee34-43f9-9071-2d9f86044235" (UID: "aadc22cb-ee34-43f9-9071-2d9f86044235"). InnerVolumeSpecName "kube-api-access-qwkdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:26:13.906811 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:13.906645    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwkdv\" (UniqueName: \"kubernetes.io/projected/aadc22cb-ee34-43f9-9071-2d9f86044235-kube-api-access-qwkdv\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:26:13.906811 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:13.906685    5093 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aadc22cb-ee34-43f9-9071-2d9f86044235-metrics-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:26:13.906811 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:13.906699    5093 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aadc22cb-ee34-43f9-9071-2d9f86044235-maas-api-tls\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:26:14.592828 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:14.592796    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e3e617-d9869f767-bmr4t"
Oct 30 13:26:14.593318 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:14.592821    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e3e617-d9869f767-bmr4t" event={"ID":"aadc22cb-ee34-43f9-9071-2d9f86044235","Type":"ContainerDied","Data":"bc287cabf015f51322dc31e30fcd9a41dd68dd70e46df78ca59795788f5b5d59"}
Oct 30 13:26:14.593318 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:14.592907    5093 scope.go:122] "RemoveContainer" containerID="afd87389dce0fea565a85a307fa2f03e83d13f15992491ba8192af962aba18db"
Oct 30 13:26:14.648541 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:14.648506    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e3e617-d9869f767-bmr4t"]
Oct 30 13:26:14.662071 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:14.662019    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e3e617-d9869f767-bmr4t"]
Oct 30 13:26:15.000266 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:15.000227    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aadc22cb-ee34-43f9-9071-2d9f86044235" path="/var/lib/kubelet/pods/aadc22cb-ee34-43f9-9071-2d9f86044235/volumes"
Oct 30 13:26:21.271623 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:26:21.271580    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:26:29.285085 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:29.285035    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-65e250-67df546b9d-xh95l"]
Oct 30 13:26:29.307110 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:29.301469    5093 cpu_manager.go:414] "containerMap: removing container" podUID="58244ace-ebd4-4b98-929b-8396655992a6" containerName="istio-proxy"
Oct 30 13:26:29.307314 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:29.307237    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58244ace-ebd4-4b98-929b-8396655992a6" containerName="istio-proxy"
Oct 30 13:26:29.307390 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:29.307321    5093 cpu_manager.go:414] "containerMap: removing container" podUID="aadc22cb-ee34-43f9-9071-2d9f86044235" containerName="maas-api"
Oct 30 13:26:29.307390 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:29.307330    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aadc22cb-ee34-43f9-9071-2d9f86044235" containerName="maas-api"
Oct 30 13:26:29.307390 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:29.307365    5093 cpu_manager.go:414] "containerMap: removing container" podUID="ca419c67-141b-4a04-8ee6-95301302c378" containerName="istio-proxy"
Oct 30 13:26:29.307390 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:29.307374    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca419c67-141b-4a04-8ee6-95301302c378" containerName="istio-proxy"
Oct 30 13:26:29.307596 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:29.307389    5093 cpu_manager.go:414] "containerMap: removing container" podUID="20970fe5-df45-4db7-8d63-6c224f8bf7bb" containerName="istio-proxy"
Oct 30 13:26:29.307596 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:29.307399    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20970fe5-df45-4db7-8d63-6c224f8bf7bb" containerName="istio-proxy"
Oct 30 13:26:29.307596 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:29.307413    5093 cpu_manager.go:414] "containerMap: removing container" podUID="ca419c67-141b-4a04-8ee6-95301302c378" containerName="istio-proxy"
Oct 30 13:26:29.307596 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:29.307421    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca419c67-141b-4a04-8ee6-95301302c378" containerName="istio-proxy"
Oct 30 13:26:29.307596 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:29.307454    5093 cpu_manager.go:414] "containerMap: removing container" podUID="8662720a-f907-42b3-a58e-d37b854a9384" containerName="maas-api"
Oct 30 13:26:29.307596 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:29.307462    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8662720a-f907-42b3-a58e-d37b854a9384" containerName="maas-api"
Oct 30 13:26:29.307596 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:29.307483    5093 cpu_manager.go:414] "containerMap: removing container" podUID="20970fe5-df45-4db7-8d63-6c224f8bf7bb" containerName="istio-proxy"
Oct 30 13:26:29.307596 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:29.307490    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20970fe5-df45-4db7-8d63-6c224f8bf7bb" containerName="istio-proxy"
Oct 30 13:26:29.307974 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:29.307742    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="aadc22cb-ee34-43f9-9071-2d9f86044235" containerName="maas-api"
Oct 30 13:26:29.307974 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:29.307775    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca419c67-141b-4a04-8ee6-95301302c378" containerName="istio-proxy"
Oct 30 13:26:29.307974 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:29.307789    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="20970fe5-df45-4db7-8d63-6c224f8bf7bb" containerName="istio-proxy"
Oct 30 13:26:29.326273 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:29.307799    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca419c67-141b-4a04-8ee6-95301302c378" containerName="istio-proxy"
Oct 30 13:26:29.326273 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:29.325219    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="8662720a-f907-42b3-a58e-d37b854a9384" containerName="maas-api"
Oct 30 13:26:29.326273 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:29.325266    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="58244ace-ebd4-4b98-929b-8396655992a6" containerName="istio-proxy"
Oct 30 13:26:29.326273 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:29.325296    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="58244ace-ebd4-4b98-929b-8396655992a6" containerName="istio-proxy"
Oct 30 13:26:29.350649 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:29.350453    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-65e250-67df546b9d-xh95l"]
Oct 30 13:26:29.350649 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:29.350562    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-65e250-67df546b9d-xh95l"
Oct 30 13:26:29.367962 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:29.367570    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-65e250-dockercfg-fkfjk\""
Oct 30 13:26:29.381157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:29.381112    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-65e250\""
Oct 30 13:26:29.447640 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:29.447595    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzqbx\" (UniqueName: \"kubernetes.io/projected/09dab442-e495-4b11-91c4-97074cd89a09-kube-api-access-kzqbx\") pod \"payload-processing-e2e-ipp-a-65e250-67df546b9d-xh95l\" (UID: \"09dab442-e495-4b11-91c4-97074cd89a09\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-65e250-67df546b9d-xh95l"
Oct 30 13:26:29.448285 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:29.448195    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09dab442-e495-4b11-91c4-97074cd89a09-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-65e250-67df546b9d-xh95l\" (UID: \"09dab442-e495-4b11-91c4-97074cd89a09\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-65e250-67df546b9d-xh95l"
Oct 30 13:26:29.549953 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:29.549862    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzqbx\" (UniqueName: \"kubernetes.io/projected/09dab442-e495-4b11-91c4-97074cd89a09-kube-api-access-kzqbx\") pod \"payload-processing-e2e-ipp-a-65e250-67df546b9d-xh95l\" (UID: \"09dab442-e495-4b11-91c4-97074cd89a09\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-65e250-67df546b9d-xh95l"
Oct 30 13:26:29.550424 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:29.550397    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09dab442-e495-4b11-91c4-97074cd89a09-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-65e250-67df546b9d-xh95l\" (UID: \"09dab442-e495-4b11-91c4-97074cd89a09\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-65e250-67df546b9d-xh95l"
Oct 30 13:26:29.551448 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:29.551420    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09dab442-e495-4b11-91c4-97074cd89a09-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-65e250-67df546b9d-xh95l\" (UID: \"09dab442-e495-4b11-91c4-97074cd89a09\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-65e250-67df546b9d-xh95l"
Oct 30 13:26:29.579191 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:29.579114    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzqbx\" (UniqueName: \"kubernetes.io/projected/09dab442-e495-4b11-91c4-97074cd89a09-kube-api-access-kzqbx\") pod \"payload-processing-e2e-ipp-a-65e250-67df546b9d-xh95l\" (UID: \"09dab442-e495-4b11-91c4-97074cd89a09\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-65e250-67df546b9d-xh95l"
Oct 30 13:26:29.669371 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:29.669337    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-65e250-67df546b9d-xh95l"
Oct 30 13:26:29.997828 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:29.997793    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-65e250-67df546b9d-xh95l"]
Oct 30 13:26:29.999592 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:26:29.999562    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09dab442_e495_4b11_91c4_97074cd89a09.slice/crio-7f889bcadf7ce663727cbc8741c59aef00c48e3d003b6b6c48cbd3b09baf7a2a WatchSource:0}: Error finding container 7f889bcadf7ce663727cbc8741c59aef00c48e3d003b6b6c48cbd3b09baf7a2a: Status 404 returned error can't find the container with id 7f889bcadf7ce663727cbc8741c59aef00c48e3d003b6b6c48cbd3b09baf7a2a
Oct 30 13:26:30.736143 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:30.736097    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-65e250-67df546b9d-xh95l" event={"ID":"09dab442-e495-4b11-91c4-97074cd89a09","Type":"ContainerStarted","Data":"99894427068fa24ed1d6e3315a76a80a75c8e639b34e2861475299dff3781e42"}
Oct 30 13:26:30.736648 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:30.736185    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-65e250-67df546b9d-xh95l"
Oct 30 13:26:30.736648 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:30.736210    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-65e250-67df546b9d-xh95l" event={"ID":"09dab442-e495-4b11-91c4-97074cd89a09","Type":"ContainerStarted","Data":"7f889bcadf7ce663727cbc8741c59aef00c48e3d003b6b6c48cbd3b09baf7a2a"}
Oct 30 13:26:30.777180 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:30.777105    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-65e250-67df546b9d-xh95l" podStartSLOduration=1.777089224 podStartE2EDuration="1.777089224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:26:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:26:30.776383037 +0000 UTC m=+1409.923650279" watchObservedRunningTime="2026-10-30 13:26:30.777089224 +0000 UTC m=+1409.924356486"
Oct 30 13:26:39.215026 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:39.214994    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb"]
Oct 30 13:26:39.215575 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:39.215263    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb" podUID="c0b3e685-671d-4227-bd45-f3a614caf44d" containerName="istio-proxy" containerID="cri-o://48061d2a25a431f7103514b13636123c2f29daa552e605f4064ee7ba4a94660b" gracePeriod=30
Oct 30 13:26:40.314880 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:40.314843    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v"]
Oct 30 13:26:40.315573 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:40.315555    5093 cpu_manager.go:414] "containerMap: removing container" podUID="58244ace-ebd4-4b98-929b-8396655992a6" containerName="istio-proxy"
Oct 30 13:26:40.315624 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:40.315575    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58244ace-ebd4-4b98-929b-8396655992a6" containerName="istio-proxy"
Oct 30 13:26:40.315708 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:40.315700    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="20970fe5-df45-4db7-8d63-6c224f8bf7bb" containerName="istio-proxy"
Oct 30 13:26:40.319525 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:40.319506    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v"
Oct 30 13:26:40.344036 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:40.344008    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v"]
Oct 30 13:26:40.351474 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:40.351447    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-ec41e5-openshift-default-dockercfg-6fkcb\""
Oct 30 13:26:40.399122 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:40.399088    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9faf395-a37d-4cae-86a5-970702e2b9f1-istio-data\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v\" (UID: \"f9faf395-a37d-4cae-86a5-970702e2b9f1\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v"
Oct 30 13:26:40.399300 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:40.399231    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9faf395-a37d-4cae-86a5-970702e2b9f1-istiod-ca-cert\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v\" (UID: \"f9faf395-a37d-4cae-86a5-970702e2b9f1\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v"
Oct 30 13:26:40.399347 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:40.399327    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9faf395-a37d-4cae-86a5-970702e2b9f1-workload-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v\" (UID: \"f9faf395-a37d-4cae-86a5-970702e2b9f1\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v"
Oct 30 13:26:40.399384 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:40.399371    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwwft\" (UniqueName: \"kubernetes.io/projected/f9faf395-a37d-4cae-86a5-970702e2b9f1-kube-api-access-zwwft\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v\" (UID: \"f9faf395-a37d-4cae-86a5-970702e2b9f1\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v"
Oct 30 13:26:40.399460 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:40.399449    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9faf395-a37d-4cae-86a5-970702e2b9f1-credential-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v\" (UID: \"f9faf395-a37d-4cae-86a5-970702e2b9f1\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v"
Oct 30 13:26:40.399519 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:40.399507    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9faf395-a37d-4cae-86a5-970702e2b9f1-istio-envoy\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v\" (UID: \"f9faf395-a37d-4cae-86a5-970702e2b9f1\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v"
Oct 30 13:26:40.399602 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:40.399539    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9faf395-a37d-4cae-86a5-970702e2b9f1-istio-podinfo\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v\" (UID: \"f9faf395-a37d-4cae-86a5-970702e2b9f1\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v"
Oct 30 13:26:40.399602 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:40.399563    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9faf395-a37d-4cae-86a5-970702e2b9f1-istio-token\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v\" (UID: \"f9faf395-a37d-4cae-86a5-970702e2b9f1\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v"
Oct 30 13:26:40.399659 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:40.399600    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9faf395-a37d-4cae-86a5-970702e2b9f1-workload-certs\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v\" (UID: \"f9faf395-a37d-4cae-86a5-970702e2b9f1\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v"
Oct 30 13:26:40.500817 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:40.500778    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9faf395-a37d-4cae-86a5-970702e2b9f1-istiod-ca-cert\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v\" (UID: \"f9faf395-a37d-4cae-86a5-970702e2b9f1\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v"
Oct 30 13:26:40.501033 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:40.500849    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9faf395-a37d-4cae-86a5-970702e2b9f1-workload-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v\" (UID: \"f9faf395-a37d-4cae-86a5-970702e2b9f1\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v"
Oct 30 13:26:40.501033 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:40.500878    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwwft\" (UniqueName: \"kubernetes.io/projected/f9faf395-a37d-4cae-86a5-970702e2b9f1-kube-api-access-zwwft\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v\" (UID: \"f9faf395-a37d-4cae-86a5-970702e2b9f1\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v"
Oct 30 13:26:40.501033 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:40.500926    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9faf395-a37d-4cae-86a5-970702e2b9f1-credential-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v\" (UID: \"f9faf395-a37d-4cae-86a5-970702e2b9f1\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v"
Oct 30 13:26:40.501033 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:40.500969    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9faf395-a37d-4cae-86a5-970702e2b9f1-istio-envoy\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v\" (UID: \"f9faf395-a37d-4cae-86a5-970702e2b9f1\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v"
Oct 30 13:26:40.501033 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:40.500999    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9faf395-a37d-4cae-86a5-970702e2b9f1-istio-podinfo\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v\" (UID: \"f9faf395-a37d-4cae-86a5-970702e2b9f1\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v"
Oct 30 13:26:40.501033 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:40.501023    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9faf395-a37d-4cae-86a5-970702e2b9f1-istio-token\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v\" (UID: \"f9faf395-a37d-4cae-86a5-970702e2b9f1\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v"
Oct 30 13:26:40.501408 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:40.501210    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9faf395-a37d-4cae-86a5-970702e2b9f1-workload-certs\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v\" (UID: \"f9faf395-a37d-4cae-86a5-970702e2b9f1\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v"
Oct 30 13:26:40.501408 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:40.501281    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9faf395-a37d-4cae-86a5-970702e2b9f1-istio-data\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v\" (UID: \"f9faf395-a37d-4cae-86a5-970702e2b9f1\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v"
Oct 30 13:26:40.501523 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:40.501449    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9faf395-a37d-4cae-86a5-970702e2b9f1-credential-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v\" (UID: \"f9faf395-a37d-4cae-86a5-970702e2b9f1\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v"
Oct 30 13:26:40.501588 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:40.501566    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9faf395-a37d-4cae-86a5-970702e2b9f1-istio-data\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v\" (UID: \"f9faf395-a37d-4cae-86a5-970702e2b9f1\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v"
Oct 30 13:26:40.501645 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:40.501624    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9faf395-a37d-4cae-86a5-970702e2b9f1-workload-certs\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v\" (UID: \"f9faf395-a37d-4cae-86a5-970702e2b9f1\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v"
Oct 30 13:26:40.501701 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:40.501652    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9faf395-a37d-4cae-86a5-970702e2b9f1-workload-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v\" (UID: \"f9faf395-a37d-4cae-86a5-970702e2b9f1\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v"
Oct 30 13:26:40.501751 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:40.501708    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9faf395-a37d-4cae-86a5-970702e2b9f1-istiod-ca-cert\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v\" (UID: \"f9faf395-a37d-4cae-86a5-970702e2b9f1\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v"
Oct 30 13:26:40.503539 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:40.503518    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9faf395-a37d-4cae-86a5-970702e2b9f1-istio-envoy\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v\" (UID: \"f9faf395-a37d-4cae-86a5-970702e2b9f1\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v"
Oct 30 13:26:40.503682 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:40.503663    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9faf395-a37d-4cae-86a5-970702e2b9f1-istio-podinfo\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v\" (UID: \"f9faf395-a37d-4cae-86a5-970702e2b9f1\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v"
Oct 30 13:26:40.521591 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:40.521558    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9faf395-a37d-4cae-86a5-970702e2b9f1-istio-token\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v\" (UID: \"f9faf395-a37d-4cae-86a5-970702e2b9f1\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v"
Oct 30 13:26:40.522392 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:40.522363    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwwft\" (UniqueName: \"kubernetes.io/projected/f9faf395-a37d-4cae-86a5-970702e2b9f1-kube-api-access-zwwft\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v\" (UID: \"f9faf395-a37d-4cae-86a5-970702e2b9f1\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v"
Oct 30 13:26:40.634228 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:40.634111    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v"
Oct 30 13:26:40.960598 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:40.960568    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v"]
Oct 30 13:26:40.962332 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:26:40.962291    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9faf395_a37d_4cae_86a5_970702e2b9f1.slice/crio-c24991acbd8d044d3a24773145e82de92b598749ff80445ecb06c3309946d628 WatchSource:0}: Error finding container c24991acbd8d044d3a24773145e82de92b598749ff80445ecb06c3309946d628: Status 404 returned error can't find the container with id c24991acbd8d044d3a24773145e82de92b598749ff80445ecb06c3309946d628
Oct 30 13:26:40.967034 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:40.966990    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:26:40.967155 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:40.967084    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:26:40.967227 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:40.967172    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:26:41.748440 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:41.748405    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-65e250-67df546b9d-xh95l"
Oct 30 13:26:41.755354 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:41.755314    5093 patch_prober.go:29] interesting pod/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:26:41.755539 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:41.755379    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb" podUID="c0b3e685-671d-4227-bd45-f3a614caf44d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:26:41.829177 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:41.825175    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v" event={"ID":"f9faf395-a37d-4cae-86a5-970702e2b9f1","Type":"ContainerStarted","Data":"ca30127e78a04dc3a047431a22a397b7f5438bd25dd44be6a007ffcd7a9efd51"}
Oct 30 13:26:41.829177 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:41.825250    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v" event={"ID":"f9faf395-a37d-4cae-86a5-970702e2b9f1","Type":"ContainerStarted","Data":"c24991acbd8d044d3a24773145e82de92b598749ff80445ecb06c3309946d628"}
Oct 30 13:26:41.901813 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:41.901756    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v" podStartSLOduration=1.901740635 podStartE2EDuration="1.901740635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:26:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:26:41.900059022 +0000 UTC m=+1421.047326255" watchObservedRunningTime="2026-10-30 13:26:41.901740635 +0000 UTC m=+1421.049007875"
Oct 30 13:26:42.640153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:42.636517    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v"
Oct 30 13:26:42.640153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:42.638370    5093 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 13:26:42.640153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:42.638443    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v" podUID="f9faf395-a37d-4cae-86a5-970702e2b9f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 13:26:43.651778 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:43.651734    5093 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 13:26:43.652399 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:43.652367    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v" podUID="f9faf395-a37d-4cae-86a5-970702e2b9f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 13:26:44.608930 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:44.608899    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb"
Oct 30 13:26:44.651613 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:44.651208    5093 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 13:26:44.651613 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:44.651267    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v" podUID="f9faf395-a37d-4cae-86a5-970702e2b9f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 13:26:44.686470 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:44.686431    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0b3e685-671d-4227-bd45-f3a614caf44d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0b3e685-671d-4227-bd45-f3a614caf44d-workload-socket\") pod \"c0b3e685-671d-4227-bd45-f3a614caf44d\" (UID: \"c0b3e685-671d-4227-bd45-f3a614caf44d\") "
Oct 30 13:26:44.687733 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:44.687697    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0b3e685-671d-4227-bd45-f3a614caf44d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0b3e685-671d-4227-bd45-f3a614caf44d-istio-podinfo\") pod \"c0b3e685-671d-4227-bd45-f3a614caf44d\" (UID: \"c0b3e685-671d-4227-bd45-f3a614caf44d\") "
Oct 30 13:26:44.687874 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:44.687793    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0b3e685-671d-4227-bd45-f3a614caf44d-kube-api-access-zhqn4\" (UniqueName: \"kubernetes.io/projected/c0b3e685-671d-4227-bd45-f3a614caf44d-kube-api-access-zhqn4\") pod \"c0b3e685-671d-4227-bd45-f3a614caf44d\" (UID: \"c0b3e685-671d-4227-bd45-f3a614caf44d\") "
Oct 30 13:26:44.687874 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:44.687826    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0b3e685-671d-4227-bd45-f3a614caf44d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0b3e685-671d-4227-bd45-f3a614caf44d-istio-data\") pod \"c0b3e685-671d-4227-bd45-f3a614caf44d\" (UID: \"c0b3e685-671d-4227-bd45-f3a614caf44d\") "
Oct 30 13:26:44.687965 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:44.687880    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0b3e685-671d-4227-bd45-f3a614caf44d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0b3e685-671d-4227-bd45-f3a614caf44d-istiod-ca-cert\") pod \"c0b3e685-671d-4227-bd45-f3a614caf44d\" (UID: \"c0b3e685-671d-4227-bd45-f3a614caf44d\") "
Oct 30 13:26:44.687965 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:44.687948    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0b3e685-671d-4227-bd45-f3a614caf44d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0b3e685-671d-4227-bd45-f3a614caf44d-workload-certs\") pod \"c0b3e685-671d-4227-bd45-f3a614caf44d\" (UID: \"c0b3e685-671d-4227-bd45-f3a614caf44d\") "
Oct 30 13:26:44.688062 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:44.688013    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0b3e685-671d-4227-bd45-f3a614caf44d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0b3e685-671d-4227-bd45-f3a614caf44d-istio-envoy\") pod \"c0b3e685-671d-4227-bd45-f3a614caf44d\" (UID: \"c0b3e685-671d-4227-bd45-f3a614caf44d\") "
Oct 30 13:26:44.688123 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:44.688102    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0b3e685-671d-4227-bd45-f3a614caf44d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0b3e685-671d-4227-bd45-f3a614caf44d-credential-socket\") pod \"c0b3e685-671d-4227-bd45-f3a614caf44d\" (UID: \"c0b3e685-671d-4227-bd45-f3a614caf44d\") "
Oct 30 13:26:44.688250 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:44.688223    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0b3e685-671d-4227-bd45-f3a614caf44d-istio-token\" (UniqueName: \"kubernetes.io/projected/c0b3e685-671d-4227-bd45-f3a614caf44d-istio-token\") pod \"c0b3e685-671d-4227-bd45-f3a614caf44d\" (UID: \"c0b3e685-671d-4227-bd45-f3a614caf44d\") "
Oct 30 13:26:44.689654 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:44.686937    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0b3e685-671d-4227-bd45-f3a614caf44d-workload-socket" pod "c0b3e685-671d-4227-bd45-f3a614caf44d" (UID: "c0b3e685-671d-4227-bd45-f3a614caf44d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:26:44.689654 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:44.688840    5093 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0b3e685-671d-4227-bd45-f3a614caf44d-workload-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:26:44.689654 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:44.689221    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0b3e685-671d-4227-bd45-f3a614caf44d-credential-socket" pod "c0b3e685-671d-4227-bd45-f3a614caf44d" (UID: "c0b3e685-671d-4227-bd45-f3a614caf44d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:26:44.689654 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:44.689533    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0b3e685-671d-4227-bd45-f3a614caf44d-istiod-ca-cert" pod "c0b3e685-671d-4227-bd45-f3a614caf44d" (UID: "c0b3e685-671d-4227-bd45-f3a614caf44d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:26:44.689654 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:44.689537    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0b3e685-671d-4227-bd45-f3a614caf44d-workload-certs" pod "c0b3e685-671d-4227-bd45-f3a614caf44d" (UID: "c0b3e685-671d-4227-bd45-f3a614caf44d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:26:44.690355 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:44.690333    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0b3e685-671d-4227-bd45-f3a614caf44d-istio-data" pod "c0b3e685-671d-4227-bd45-f3a614caf44d" (UID: "c0b3e685-671d-4227-bd45-f3a614caf44d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:26:44.692281 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:44.692260    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0b3e685-671d-4227-bd45-f3a614caf44d-kube-api-access-zhqn4" pod "c0b3e685-671d-4227-bd45-f3a614caf44d" (UID: "c0b3e685-671d-4227-bd45-f3a614caf44d"). InnerVolumeSpecName "kube-api-access-zhqn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:26:44.693590 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:44.693560    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0b3e685-671d-4227-bd45-f3a614caf44d-istio-podinfo" pod "c0b3e685-671d-4227-bd45-f3a614caf44d" (UID: "c0b3e685-671d-4227-bd45-f3a614caf44d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:26:44.694879 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:44.694858    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0b3e685-671d-4227-bd45-f3a614caf44d-istio-token" pod "c0b3e685-671d-4227-bd45-f3a614caf44d" (UID: "c0b3e685-671d-4227-bd45-f3a614caf44d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:26:44.694988 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:44.694913    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0b3e685-671d-4227-bd45-f3a614caf44d-istio-envoy" pod "c0b3e685-671d-4227-bd45-f3a614caf44d" (UID: "c0b3e685-671d-4227-bd45-f3a614caf44d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:26:44.790405 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:44.790357    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhqn4\" (UniqueName: \"kubernetes.io/projected/c0b3e685-671d-4227-bd45-f3a614caf44d-kube-api-access-zhqn4\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:26:44.790405 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:44.790402    5093 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0b3e685-671d-4227-bd45-f3a614caf44d-istio-data\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:26:44.790405 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:44.790413    5093 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0b3e685-671d-4227-bd45-f3a614caf44d-istiod-ca-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:26:44.790645 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:44.790423    5093 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0b3e685-671d-4227-bd45-f3a614caf44d-workload-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:26:44.790645 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:44.790431    5093 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0b3e685-671d-4227-bd45-f3a614caf44d-istio-envoy\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:26:44.790645 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:44.790454    5093 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0b3e685-671d-4227-bd45-f3a614caf44d-credential-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:26:44.790645 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:44.790467    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0b3e685-671d-4227-bd45-f3a614caf44d-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:26:44.790645 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:44.790476    5093 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0b3e685-671d-4227-bd45-f3a614caf44d-istio-podinfo\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:26:44.878068 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:44.877981    5093 generic.go:356] "Generic (PLEG): container finished" podID="c0b3e685-671d-4227-bd45-f3a614caf44d" containerID="48061d2a25a431f7103514b13636123c2f29daa552e605f4064ee7ba4a94660b" exitCode=0
Oct 30 13:26:44.878429 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:44.878284    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb" event={"ID":"c0b3e685-671d-4227-bd45-f3a614caf44d","Type":"ContainerDied","Data":"48061d2a25a431f7103514b13636123c2f29daa552e605f4064ee7ba4a94660b"}
Oct 30 13:26:44.878593 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:44.878582    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb" event={"ID":"c0b3e685-671d-4227-bd45-f3a614caf44d","Type":"ContainerDied","Data":"b445cdc7ace3ddff0e25b2a56145532416e51cef4d495435c89f19f724aabecf"}
Oct 30 13:26:44.878698 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:44.878667    5093 scope.go:122] "RemoveContainer" containerID="48061d2a25a431f7103514b13636123c2f29daa552e605f4064ee7ba4a94660b"
Oct 30 13:26:44.878844 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:44.878311    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb"
Oct 30 13:26:44.891495 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:44.891474    5093 scope.go:122] "RemoveContainer" containerID="941f9890e9b596c68bee2804faf8cf39e419581aac827f00c4f546dd06362a9b"
Oct 30 13:26:44.901801 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:44.901778    5093 scope.go:122] "RemoveContainer" containerID="48061d2a25a431f7103514b13636123c2f29daa552e605f4064ee7ba4a94660b"
Oct 30 13:26:44.902113 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:26:44.902084    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48061d2a25a431f7103514b13636123c2f29daa552e605f4064ee7ba4a94660b\": container with ID starting with 48061d2a25a431f7103514b13636123c2f29daa552e605f4064ee7ba4a94660b not found: ID does not exist" containerID="48061d2a25a431f7103514b13636123c2f29daa552e605f4064ee7ba4a94660b"
Oct 30 13:26:44.902287 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:44.902125    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48061d2a25a431f7103514b13636123c2f29daa552e605f4064ee7ba4a94660b"} err="failed to get container status \"48061d2a25a431f7103514b13636123c2f29daa552e605f4064ee7ba4a94660b\": rpc error: code = NotFound desc = could not find container \"48061d2a25a431f7103514b13636123c2f29daa552e605f4064ee7ba4a94660b\": container with ID starting with 48061d2a25a431f7103514b13636123c2f29daa552e605f4064ee7ba4a94660b not found: ID does not exist"
Oct 30 13:26:44.902287 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:44.902170    5093 scope.go:122] "RemoveContainer" containerID="941f9890e9b596c68bee2804faf8cf39e419581aac827f00c4f546dd06362a9b"
Oct 30 13:26:44.902450 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:26:44.902430    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"941f9890e9b596c68bee2804faf8cf39e419581aac827f00c4f546dd06362a9b\": container with ID starting with 941f9890e9b596c68bee2804faf8cf39e419581aac827f00c4f546dd06362a9b not found: ID does not exist" containerID="941f9890e9b596c68bee2804faf8cf39e419581aac827f00c4f546dd06362a9b"
Oct 30 13:26:44.902488 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:44.902460    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"941f9890e9b596c68bee2804faf8cf39e419581aac827f00c4f546dd06362a9b"} err="failed to get container status \"941f9890e9b596c68bee2804faf8cf39e419581aac827f00c4f546dd06362a9b\": rpc error: code = NotFound desc = could not find container \"941f9890e9b596c68bee2804faf8cf39e419581aac827f00c4f546dd06362a9b\": container with ID starting with 941f9890e9b596c68bee2804faf8cf39e419581aac827f00c4f546dd06362a9b not found: ID does not exist"
Oct 30 13:26:44.964360 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:44.964309    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb"]
Oct 30 13:26:44.969938 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:44.969905    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-8ac28e-openshift-default-66bcb64489-m5lzb"]
Oct 30 13:26:44.999444 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:44.999412    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0b3e685-671d-4227-bd45-f3a614caf44d" path="/var/lib/kubelet/pods/c0b3e685-671d-4227-bd45-f3a614caf44d/volumes"
Oct 30 13:26:45.266163 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:45.266107    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-65e250-58c5679c7-lbn2k"]
Oct 30 13:26:45.266926 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:45.266900    5093 cpu_manager.go:414] "containerMap: removing container" podUID="c0b3e685-671d-4227-bd45-f3a614caf44d" containerName="istio-proxy"
Oct 30 13:26:45.266926 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:45.266923    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0b3e685-671d-4227-bd45-f3a614caf44d" containerName="istio-proxy"
Oct 30 13:26:45.267114 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:45.266942    5093 cpu_manager.go:414] "containerMap: removing container" podUID="c0b3e685-671d-4227-bd45-f3a614caf44d" containerName="istio-proxy"
Oct 30 13:26:45.267114 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:45.266951    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0b3e685-671d-4227-bd45-f3a614caf44d" containerName="istio-proxy"
Oct 30 13:26:45.267249 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:45.267188    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0b3e685-671d-4227-bd45-f3a614caf44d" containerName="istio-proxy"
Oct 30 13:26:45.267249 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:45.267209    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0b3e685-671d-4227-bd45-f3a614caf44d" containerName="istio-proxy"
Oct 30 13:26:45.289179 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:45.283308    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-65e250-58c5679c7-lbn2k"
Oct 30 13:26:45.289179 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:45.288249    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-65e250-58c5679c7-lbn2k"]
Oct 30 13:26:45.305163 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:45.304241    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-65e250-serving-cert\""
Oct 30 13:26:45.398794 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:45.398751    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2z2b\" (UniqueName: \"kubernetes.io/projected/6b4e2373-e7d6-451d-9213-257c6b56ef16-kube-api-access-p2z2b\") pod \"maas-api-e2e-ipp-a-65e250-58c5679c7-lbn2k\" (UID: \"6b4e2373-e7d6-451d-9213-257c6b56ef16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-65e250-58c5679c7-lbn2k"
Oct 30 13:26:45.398974 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:45.398820    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b4e2373-e7d6-451d-9213-257c6b56ef16-metrics-certs\") pod \"maas-api-e2e-ipp-a-65e250-58c5679c7-lbn2k\" (UID: \"6b4e2373-e7d6-451d-9213-257c6b56ef16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-65e250-58c5679c7-lbn2k"
Oct 30 13:26:45.398974 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:45.398926    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b4e2373-e7d6-451d-9213-257c6b56ef16-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-65e250-58c5679c7-lbn2k\" (UID: \"6b4e2373-e7d6-451d-9213-257c6b56ef16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-65e250-58c5679c7-lbn2k"
Oct 30 13:26:45.399107 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:45.398982    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b4e2373-e7d6-451d-9213-257c6b56ef16-maas-api-tls\") pod \"maas-api-e2e-ipp-a-65e250-58c5679c7-lbn2k\" (UID: \"6b4e2373-e7d6-451d-9213-257c6b56ef16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-65e250-58c5679c7-lbn2k"
Oct 30 13:26:45.500625 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:45.500586    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b4e2373-e7d6-451d-9213-257c6b56ef16-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-65e250-58c5679c7-lbn2k\" (UID: \"6b4e2373-e7d6-451d-9213-257c6b56ef16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-65e250-58c5679c7-lbn2k"
Oct 30 13:26:45.500827 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:45.500690    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b4e2373-e7d6-451d-9213-257c6b56ef16-maas-api-tls\") pod \"maas-api-e2e-ipp-a-65e250-58c5679c7-lbn2k\" (UID: \"6b4e2373-e7d6-451d-9213-257c6b56ef16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-65e250-58c5679c7-lbn2k"
Oct 30 13:26:45.500827 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:45.500774    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2z2b\" (UniqueName: \"kubernetes.io/projected/6b4e2373-e7d6-451d-9213-257c6b56ef16-kube-api-access-p2z2b\") pod \"maas-api-e2e-ipp-a-65e250-58c5679c7-lbn2k\" (UID: \"6b4e2373-e7d6-451d-9213-257c6b56ef16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-65e250-58c5679c7-lbn2k"
Oct 30 13:26:45.500949 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:45.500835    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b4e2373-e7d6-451d-9213-257c6b56ef16-metrics-certs\") pod \"maas-api-e2e-ipp-a-65e250-58c5679c7-lbn2k\" (UID: \"6b4e2373-e7d6-451d-9213-257c6b56ef16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-65e250-58c5679c7-lbn2k"
Oct 30 13:26:45.501307 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:45.501284    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b4e2373-e7d6-451d-9213-257c6b56ef16-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-65e250-58c5679c7-lbn2k\" (UID: \"6b4e2373-e7d6-451d-9213-257c6b56ef16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-65e250-58c5679c7-lbn2k"
Oct 30 13:26:45.503251 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:45.503229    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b4e2373-e7d6-451d-9213-257c6b56ef16-metrics-certs\") pod \"maas-api-e2e-ipp-a-65e250-58c5679c7-lbn2k\" (UID: \"6b4e2373-e7d6-451d-9213-257c6b56ef16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-65e250-58c5679c7-lbn2k"
Oct 30 13:26:45.503349 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:45.503290    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b4e2373-e7d6-451d-9213-257c6b56ef16-maas-api-tls\") pod \"maas-api-e2e-ipp-a-65e250-58c5679c7-lbn2k\" (UID: \"6b4e2373-e7d6-451d-9213-257c6b56ef16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-65e250-58c5679c7-lbn2k"
Oct 30 13:26:45.520767 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:45.520694    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2z2b\" (UniqueName: \"kubernetes.io/projected/6b4e2373-e7d6-451d-9213-257c6b56ef16-kube-api-access-p2z2b\") pod \"maas-api-e2e-ipp-a-65e250-58c5679c7-lbn2k\" (UID: \"6b4e2373-e7d6-451d-9213-257c6b56ef16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-65e250-58c5679c7-lbn2k"
Oct 30 13:26:45.605410 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:45.605375    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-65e250-58c5679c7-lbn2k"
Oct 30 13:26:45.671076 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:45.670598    5093 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 13:26:45.671076 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:45.670669    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v" podUID="f9faf395-a37d-4cae-86a5-970702e2b9f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 13:26:45.970317 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:45.970153    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-65e250-58c5679c7-lbn2k"]
Oct 30 13:26:46.650219 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:46.646558    5093 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 13:26:46.650219 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:46.646627    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v" podUID="f9faf395-a37d-4cae-86a5-970702e2b9f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 13:26:46.902262 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:46.902162    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-65e250-58c5679c7-lbn2k" event={"ID":"6b4e2373-e7d6-451d-9213-257c6b56ef16","Type":"ContainerStarted","Data":"3d0548fb72044a63d6587fa6c21562a6b4dd5a4d574e8e7b9dece7142509240a"}
Oct 30 13:26:46.902262 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:46.902231    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-65e250-58c5679c7-lbn2k" event={"ID":"6b4e2373-e7d6-451d-9213-257c6b56ef16","Type":"ContainerStarted","Data":"cf0c6ff89a5f32c33056d17b7ba8acc16efc5f4e1ddebbc5866a0fc55ac1fef0"}
Oct 30 13:26:46.902262 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:46.902257    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-65e250-58c5679c7-lbn2k"
Oct 30 13:26:47.634846 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:47.634813    5093 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 13:26:47.635067 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:47.634867    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v" podUID="f9faf395-a37d-4cae-86a5-970702e2b9f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 13:26:48.827001 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:48.826969    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v"
Oct 30 13:26:48.827858 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:48.827836    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v"
Oct 30 13:26:48.828711 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:48.828687    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v"
Oct 30 13:26:48.898062 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:48.898007    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-65e250-58c5679c7-lbn2k" podStartSLOduration=3.49330852 podStartE2EDuration="3.897983286s" totalImagesPullingTime="404.674766ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:26:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:26:46.95690432 +0000 UTC m=+1426.104171562" watchObservedRunningTime="2026-10-30 13:26:48.897983286 +0000 UTC m=+1428.045250526"
Oct 30 13:26:51.170247 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:26:51.170209    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:26:52.918953 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:52.918920    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-65e250-58c5679c7-lbn2k"
Oct 30 13:26:55.378890 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.378851    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq"]
Oct 30 13:26:55.383408 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.383387    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq"
Oct 30 13:26:55.399765 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.399738    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5"]
Oct 30 13:26:55.403854 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.403835    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5"
Oct 30 13:26:55.415787 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.415760    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq"]
Oct 30 13:26:55.432647 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.432611    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5"]
Oct 30 13:26:55.497522 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.497488    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f16ce08-b99b-4843-9213-9f78ac72626a-istio-envoy\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq\" (UID: \"7f16ce08-b99b-4843-9213-9f78ac72626a\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq"
Oct 30 13:26:55.497671 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.497552    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94dnx\" (UniqueName: \"kubernetes.io/projected/7f16ce08-b99b-4843-9213-9f78ac72626a-kube-api-access-94dnx\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq\" (UID: \"7f16ce08-b99b-4843-9213-9f78ac72626a\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq"
Oct 30 13:26:55.497708 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.497678    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f16ce08-b99b-4843-9213-9f78ac72626a-workload-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq\" (UID: \"7f16ce08-b99b-4843-9213-9f78ac72626a\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq"
Oct 30 13:26:55.497833 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.497814    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f16ce08-b99b-4843-9213-9f78ac72626a-istiod-ca-cert\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq\" (UID: \"7f16ce08-b99b-4843-9213-9f78ac72626a\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq"
Oct 30 13:26:55.497870 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.497854    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f16ce08-b99b-4843-9213-9f78ac72626a-istio-podinfo\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq\" (UID: \"7f16ce08-b99b-4843-9213-9f78ac72626a\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq"
Oct 30 13:26:55.497905 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.497874    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f16ce08-b99b-4843-9213-9f78ac72626a-istio-token\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq\" (UID: \"7f16ce08-b99b-4843-9213-9f78ac72626a\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq"
Oct 30 13:26:55.497991 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.497974    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f16ce08-b99b-4843-9213-9f78ac72626a-istio-data\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq\" (UID: \"7f16ce08-b99b-4843-9213-9f78ac72626a\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq"
Oct 30 13:26:55.498037 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.498024    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f16ce08-b99b-4843-9213-9f78ac72626a-credential-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq\" (UID: \"7f16ce08-b99b-4843-9213-9f78ac72626a\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq"
Oct 30 13:26:55.498095 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.498082    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f16ce08-b99b-4843-9213-9f78ac72626a-workload-certs\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq\" (UID: \"7f16ce08-b99b-4843-9213-9f78ac72626a\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq"
Oct 30 13:26:55.599193 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.599158    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b57be897-da31-47b5-bf14-5a11969f94d0-credential-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5\" (UID: \"b57be897-da31-47b5-bf14-5a11969f94d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5"
Oct 30 13:26:55.599373 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.599228    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94dnx\" (UniqueName: \"kubernetes.io/projected/7f16ce08-b99b-4843-9213-9f78ac72626a-kube-api-access-94dnx\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq\" (UID: \"7f16ce08-b99b-4843-9213-9f78ac72626a\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq"
Oct 30 13:26:55.599373 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.599248    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b57be897-da31-47b5-bf14-5a11969f94d0-workload-certs\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5\" (UID: \"b57be897-da31-47b5-bf14-5a11969f94d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5"
Oct 30 13:26:55.599373 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.599282    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f16ce08-b99b-4843-9213-9f78ac72626a-workload-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq\" (UID: \"7f16ce08-b99b-4843-9213-9f78ac72626a\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq"
Oct 30 13:26:55.599373 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.599304    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b57be897-da31-47b5-bf14-5a11969f94d0-workload-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5\" (UID: \"b57be897-da31-47b5-bf14-5a11969f94d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5"
Oct 30 13:26:55.599530 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.599486    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b57be897-da31-47b5-bf14-5a11969f94d0-istio-token\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5\" (UID: \"b57be897-da31-47b5-bf14-5a11969f94d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5"
Oct 30 13:26:55.599575 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.599532    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b57be897-da31-47b5-bf14-5a11969f94d0-istio-envoy\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5\" (UID: \"b57be897-da31-47b5-bf14-5a11969f94d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5"
Oct 30 13:26:55.599701 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.599678    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwdd5\" (UniqueName: \"kubernetes.io/projected/b57be897-da31-47b5-bf14-5a11969f94d0-kube-api-access-bwdd5\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5\" (UID: \"b57be897-da31-47b5-bf14-5a11969f94d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5"
Oct 30 13:26:55.599756 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.599740    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b57be897-da31-47b5-bf14-5a11969f94d0-istio-data\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5\" (UID: \"b57be897-da31-47b5-bf14-5a11969f94d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5"
Oct 30 13:26:55.599810 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.599778    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b57be897-da31-47b5-bf14-5a11969f94d0-istio-podinfo\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5\" (UID: \"b57be897-da31-47b5-bf14-5a11969f94d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5"
Oct 30 13:26:55.599810 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.599780    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f16ce08-b99b-4843-9213-9f78ac72626a-workload-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq\" (UID: \"7f16ce08-b99b-4843-9213-9f78ac72626a\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq"
Oct 30 13:26:55.599912 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.599866    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f16ce08-b99b-4843-9213-9f78ac72626a-istiod-ca-cert\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq\" (UID: \"7f16ce08-b99b-4843-9213-9f78ac72626a\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq"
Oct 30 13:26:55.599969 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.599930    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f16ce08-b99b-4843-9213-9f78ac72626a-istio-podinfo\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq\" (UID: \"7f16ce08-b99b-4843-9213-9f78ac72626a\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq"
Oct 30 13:26:55.599969 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.599959    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f16ce08-b99b-4843-9213-9f78ac72626a-istio-token\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq\" (UID: \"7f16ce08-b99b-4843-9213-9f78ac72626a\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq"
Oct 30 13:26:55.600068 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.600011    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b57be897-da31-47b5-bf14-5a11969f94d0-istiod-ca-cert\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5\" (UID: \"b57be897-da31-47b5-bf14-5a11969f94d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5"
Oct 30 13:26:55.600068 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.600048    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f16ce08-b99b-4843-9213-9f78ac72626a-istio-data\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq\" (UID: \"7f16ce08-b99b-4843-9213-9f78ac72626a\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq"
Oct 30 13:26:55.600216 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.600107    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f16ce08-b99b-4843-9213-9f78ac72626a-credential-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq\" (UID: \"7f16ce08-b99b-4843-9213-9f78ac72626a\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq"
Oct 30 13:26:55.600216 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.600185    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f16ce08-b99b-4843-9213-9f78ac72626a-workload-certs\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq\" (UID: \"7f16ce08-b99b-4843-9213-9f78ac72626a\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq"
Oct 30 13:26:55.600442 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.600274    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f16ce08-b99b-4843-9213-9f78ac72626a-istio-envoy\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq\" (UID: \"7f16ce08-b99b-4843-9213-9f78ac72626a\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq"
Oct 30 13:26:55.600442 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.600391    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f16ce08-b99b-4843-9213-9f78ac72626a-istio-data\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq\" (UID: \"7f16ce08-b99b-4843-9213-9f78ac72626a\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq"
Oct 30 13:26:55.600571 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.600549    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f16ce08-b99b-4843-9213-9f78ac72626a-istiod-ca-cert\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq\" (UID: \"7f16ce08-b99b-4843-9213-9f78ac72626a\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq"
Oct 30 13:26:55.600571 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.600557    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f16ce08-b99b-4843-9213-9f78ac72626a-credential-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq\" (UID: \"7f16ce08-b99b-4843-9213-9f78ac72626a\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq"
Oct 30 13:26:55.600922 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.600899    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f16ce08-b99b-4843-9213-9f78ac72626a-workload-certs\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq\" (UID: \"7f16ce08-b99b-4843-9213-9f78ac72626a\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq"
Oct 30 13:26:55.602788 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.602762    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f16ce08-b99b-4843-9213-9f78ac72626a-istio-envoy\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq\" (UID: \"7f16ce08-b99b-4843-9213-9f78ac72626a\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq"
Oct 30 13:26:55.603064 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.603035    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f16ce08-b99b-4843-9213-9f78ac72626a-istio-podinfo\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq\" (UID: \"7f16ce08-b99b-4843-9213-9f78ac72626a\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq"
Oct 30 13:26:55.621281 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.621247    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94dnx\" (UniqueName: \"kubernetes.io/projected/7f16ce08-b99b-4843-9213-9f78ac72626a-kube-api-access-94dnx\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq\" (UID: \"7f16ce08-b99b-4843-9213-9f78ac72626a\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq"
Oct 30 13:26:55.623813 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.623785    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f16ce08-b99b-4843-9213-9f78ac72626a-istio-token\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq\" (UID: \"7f16ce08-b99b-4843-9213-9f78ac72626a\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq"
Oct 30 13:26:55.699881 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.699836    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq"
Oct 30 13:26:55.701399 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.701365    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b57be897-da31-47b5-bf14-5a11969f94d0-credential-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5\" (UID: \"b57be897-da31-47b5-bf14-5a11969f94d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5"
Oct 30 13:26:55.701541 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.701440    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b57be897-da31-47b5-bf14-5a11969f94d0-workload-certs\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5\" (UID: \"b57be897-da31-47b5-bf14-5a11969f94d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5"
Oct 30 13:26:55.701541 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.701493    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b57be897-da31-47b5-bf14-5a11969f94d0-workload-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5\" (UID: \"b57be897-da31-47b5-bf14-5a11969f94d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5"
Oct 30 13:26:55.701664 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.701549    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b57be897-da31-47b5-bf14-5a11969f94d0-istio-token\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5\" (UID: \"b57be897-da31-47b5-bf14-5a11969f94d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5"
Oct 30 13:26:55.701664 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.701580    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b57be897-da31-47b5-bf14-5a11969f94d0-istio-envoy\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5\" (UID: \"b57be897-da31-47b5-bf14-5a11969f94d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5"
Oct 30 13:26:55.701664 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.701649    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwdd5\" (UniqueName: \"kubernetes.io/projected/b57be897-da31-47b5-bf14-5a11969f94d0-kube-api-access-bwdd5\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5\" (UID: \"b57be897-da31-47b5-bf14-5a11969f94d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5"
Oct 30 13:26:55.701820 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.701699    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b57be897-da31-47b5-bf14-5a11969f94d0-istio-data\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5\" (UID: \"b57be897-da31-47b5-bf14-5a11969f94d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5"
Oct 30 13:26:55.701820 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.701732    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b57be897-da31-47b5-bf14-5a11969f94d0-istio-podinfo\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5\" (UID: \"b57be897-da31-47b5-bf14-5a11969f94d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5"
Oct 30 13:26:55.701820 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.701761    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b57be897-da31-47b5-bf14-5a11969f94d0-credential-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5\" (UID: \"b57be897-da31-47b5-bf14-5a11969f94d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5"
Oct 30 13:26:55.701820 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.701815    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b57be897-da31-47b5-bf14-5a11969f94d0-istiod-ca-cert\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5\" (UID: \"b57be897-da31-47b5-bf14-5a11969f94d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5"
Oct 30 13:26:55.702022 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.701865    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b57be897-da31-47b5-bf14-5a11969f94d0-workload-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5\" (UID: \"b57be897-da31-47b5-bf14-5a11969f94d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5"
Oct 30 13:26:55.702022 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.701814    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b57be897-da31-47b5-bf14-5a11969f94d0-workload-certs\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5\" (UID: \"b57be897-da31-47b5-bf14-5a11969f94d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5"
Oct 30 13:26:55.702238 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.702215    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b57be897-da31-47b5-bf14-5a11969f94d0-istio-data\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5\" (UID: \"b57be897-da31-47b5-bf14-5a11969f94d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5"
Oct 30 13:26:55.702600 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.702581    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b57be897-da31-47b5-bf14-5a11969f94d0-istiod-ca-cert\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5\" (UID: \"b57be897-da31-47b5-bf14-5a11969f94d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5"
Oct 30 13:26:55.704095 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.704048    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b57be897-da31-47b5-bf14-5a11969f94d0-istio-envoy\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5\" (UID: \"b57be897-da31-47b5-bf14-5a11969f94d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5"
Oct 30 13:26:55.704233 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.704111    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b57be897-da31-47b5-bf14-5a11969f94d0-istio-podinfo\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5\" (UID: \"b57be897-da31-47b5-bf14-5a11969f94d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5"
Oct 30 13:26:55.723701 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.723671    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b57be897-da31-47b5-bf14-5a11969f94d0-istio-token\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5\" (UID: \"b57be897-da31-47b5-bf14-5a11969f94d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5"
Oct 30 13:26:55.724239 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:55.724217    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwdd5\" (UniqueName: \"kubernetes.io/projected/b57be897-da31-47b5-bf14-5a11969f94d0-kube-api-access-bwdd5\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5\" (UID: \"b57be897-da31-47b5-bf14-5a11969f94d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5"
Oct 30 13:26:56.020331 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:56.020245    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5"
Oct 30 13:26:56.030720 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:56.030690    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq"]
Oct 30 13:26:56.033634 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:26:56.033602    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f16ce08_b99b_4843_9213_9f78ac72626a.slice/crio-4021fb81498c8f98199e7866fa13a992fef7d1e1104da33698d926a1b27a75e5 WatchSource:0}: Error finding container 4021fb81498c8f98199e7866fa13a992fef7d1e1104da33698d926a1b27a75e5: Status 404 returned error can't find the container with id 4021fb81498c8f98199e7866fa13a992fef7d1e1104da33698d926a1b27a75e5
Oct 30 13:26:56.037314 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:56.037272    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:26:56.037418 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:56.037365    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:26:56.037418 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:56.037409    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:26:56.353808 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:56.353773    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5"]
Oct 30 13:26:56.354290 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:26:56.354255    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb57be897_da31_47b5_bf14_5a11969f94d0.slice/crio-3137bba8025fcafbce26be2425fe09ed94d7e9b109fb2a11bc69b08cd36791bc WatchSource:0}: Error finding container 3137bba8025fcafbce26be2425fe09ed94d7e9b109fb2a11bc69b08cd36791bc: Status 404 returned error can't find the container with id 3137bba8025fcafbce26be2425fe09ed94d7e9b109fb2a11bc69b08cd36791bc
Oct 30 13:26:56.357458 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:56.357419    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:26:56.357536 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:56.357508    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:26:56.357587 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:56.357552    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:26:56.986229 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:56.986190    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq" event={"ID":"7f16ce08-b99b-4843-9213-9f78ac72626a","Type":"ContainerStarted","Data":"a82f07ff4230092f7b6607dbb5090ef02ae433e323b213a45628f6ec44fc3c84"}
Oct 30 13:26:56.986677 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:56.986256    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq" event={"ID":"7f16ce08-b99b-4843-9213-9f78ac72626a","Type":"ContainerStarted","Data":"4021fb81498c8f98199e7866fa13a992fef7d1e1104da33698d926a1b27a75e5"}
Oct 30 13:26:57.000626 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:57.000590    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5" event={"ID":"b57be897-da31-47b5-bf14-5a11969f94d0","Type":"ContainerStarted","Data":"8e8edc539375a5e71330d33108ebf64346727cb1db108d9c180d4ab554c1fcc2"}
Oct 30 13:26:57.000777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:57.000651    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5" event={"ID":"b57be897-da31-47b5-bf14-5a11969f94d0","Type":"ContainerStarted","Data":"3137bba8025fcafbce26be2425fe09ed94d7e9b109fb2a11bc69b08cd36791bc"}
Oct 30 13:26:57.020952 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:57.020918    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5"
Oct 30 13:26:57.022614 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:57.022582    5093 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 13:26:57.022743 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:57.022640    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5" podUID="b57be897-da31-47b5-bf14-5a11969f94d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 13:26:57.038556 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:57.038492    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq" podStartSLOduration=2.038473939 podStartE2EDuration="2.038473939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:26:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:26:57.033591435 +0000 UTC m=+1436.180858678" watchObservedRunningTime="2026-10-30 13:26:57.038473939 +0000 UTC m=+1436.185741182"
Oct 30 13:26:57.097658 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:57.097593    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5" podStartSLOduration=2.097574139 podStartE2EDuration="2.097574139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:26:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:26:57.095822459 +0000 UTC m=+1436.243089693" watchObservedRunningTime="2026-10-30 13:26:57.097574139 +0000 UTC m=+1436.244841422"
Oct 30 13:26:57.700059 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:57.700012    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq"
Oct 30 13:26:57.704264 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:57.704232    5093 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 13:26:57.704454 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:57.704295    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq" podUID="7f16ce08-b99b-4843-9213-9f78ac72626a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 13:26:58.021035 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:58.020950    5093 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 13:26:58.021035 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:58.021002    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5" podUID="b57be897-da31-47b5-bf14-5a11969f94d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 13:26:58.700218 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:58.700182    5093 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 13:26:58.700402 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:58.700235    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq" podUID="7f16ce08-b99b-4843-9213-9f78ac72626a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 13:26:59.021512 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:59.021421    5093 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 13:26:59.021512 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:59.021486    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5" podUID="b57be897-da31-47b5-bf14-5a11969f94d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 13:26:59.700485 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:59.700446    5093 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 13:26:59.700705 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:26:59.700519    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq" podUID="7f16ce08-b99b-4843-9213-9f78ac72626a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 13:27:00.020735 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:00.020641    5093 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 13:27:00.023892 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:00.020707    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5" podUID="b57be897-da31-47b5-bf14-5a11969f94d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 13:27:00.712301 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:00.708969    5093 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 13:27:00.712301 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:00.709036    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq" podUID="7f16ce08-b99b-4843-9213-9f78ac72626a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 13:27:01.020712 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:01.020628    5093 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 13:27:01.020712 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:01.020696    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5" podUID="b57be897-da31-47b5-bf14-5a11969f94d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 13:27:01.549049 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:01.549006    5093 scope.go:122] "RemoveContainer" containerID="076cca7837bb30d2da69f3f35ee8eeb3ee8ca253ba90ec04f6f82999128e229b"
Oct 30 13:27:01.700411 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:01.700368    5093 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 13:27:01.700627 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:01.700443    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq" podUID="7f16ce08-b99b-4843-9213-9f78ac72626a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 13:27:02.021124 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:02.021085    5093 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 13:27:02.021359 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:02.021176    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5" podUID="b57be897-da31-47b5-bf14-5a11969f94d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 13:27:02.700953 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:02.700918    5093 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 13:27:02.701399 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:02.700989    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq" podUID="7f16ce08-b99b-4843-9213-9f78ac72626a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 13:27:03.021266 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:03.021156    5093 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 13:27:03.021266 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:03.021223    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5" podUID="b57be897-da31-47b5-bf14-5a11969f94d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 13:27:03.700679 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:03.700632    5093 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 13:27:03.700865 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:03.700707    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq" podUID="7f16ce08-b99b-4843-9213-9f78ac72626a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 13:27:04.021648 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:04.021547    5093 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 13:27:04.021648 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:04.021631    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5" podUID="b57be897-da31-47b5-bf14-5a11969f94d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 13:27:04.747150 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:04.745723    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq"
Oct 30 13:27:04.747150 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:04.746820    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq"
Oct 30 13:27:04.770543 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:04.770460    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq"
Oct 30 13:27:05.581985 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:05.581413    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5"
Oct 30 13:27:05.582707 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:05.582433    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5"
Oct 30 13:27:05.583707 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:05.583679    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5"
Oct 30 13:27:13.360626 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:13.360581    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-65e250-67df546b9d-xh95l"]
Oct 30 13:27:18.361241 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:18.361183    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-65e250-67df546b9d-xh95l" podUID="09dab442-e495-4b11-91c4-97074cd89a09" containerName="payload-processing" containerID="cri-o://99894427068fa24ed1d6e3315a76a80a75c8e639b34e2861475299dff3781e42" gracePeriod=25
Oct 30 13:27:18.685080 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:18.685050    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-65e250-67df546b9d-xh95l"
Oct 30 13:27:18.787937 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:18.787896    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09dab442-e495-4b11-91c4-97074cd89a09-kube-api-access-kzqbx\" (UniqueName: \"kubernetes.io/projected/09dab442-e495-4b11-91c4-97074cd89a09-kube-api-access-kzqbx\") pod \"09dab442-e495-4b11-91c4-97074cd89a09\" (UID: \"09dab442-e495-4b11-91c4-97074cd89a09\") "
Oct 30 13:27:18.788105 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:18.788054    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09dab442-e495-4b11-91c4-97074cd89a09-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09dab442-e495-4b11-91c4-97074cd89a09-plugins-config-volume\") pod \"09dab442-e495-4b11-91c4-97074cd89a09\" (UID: \"09dab442-e495-4b11-91c4-97074cd89a09\") "
Oct 30 13:27:18.788443 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:18.788414    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09dab442-e495-4b11-91c4-97074cd89a09-plugins-config-volume" pod "09dab442-e495-4b11-91c4-97074cd89a09" (UID: "09dab442-e495-4b11-91c4-97074cd89a09"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:27:18.788772 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:18.788746    5093 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09dab442-e495-4b11-91c4-97074cd89a09-plugins-config-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:27:18.792922 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:18.792896    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09dab442-e495-4b11-91c4-97074cd89a09-kube-api-access-kzqbx" pod "09dab442-e495-4b11-91c4-97074cd89a09" (UID: "09dab442-e495-4b11-91c4-97074cd89a09"). InnerVolumeSpecName "kube-api-access-kzqbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:27:18.890823 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:18.890783    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzqbx\" (UniqueName: \"kubernetes.io/projected/09dab442-e495-4b11-91c4-97074cd89a09-kube-api-access-kzqbx\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:27:19.490860 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:19.490820    5093 generic.go:356] "Generic (PLEG): container finished" podID="09dab442-e495-4b11-91c4-97074cd89a09" containerID="99894427068fa24ed1d6e3315a76a80a75c8e639b34e2861475299dff3781e42" exitCode=0
Oct 30 13:27:19.490860 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:19.490891    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-65e250-67df546b9d-xh95l"
Oct 30 13:27:19.490860 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:19.490920    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-65e250-67df546b9d-xh95l" event={"ID":"09dab442-e495-4b11-91c4-97074cd89a09","Type":"ContainerDied","Data":"99894427068fa24ed1d6e3315a76a80a75c8e639b34e2861475299dff3781e42"}
Oct 30 13:27:19.491635 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:19.491004    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-65e250-67df546b9d-xh95l" event={"ID":"09dab442-e495-4b11-91c4-97074cd89a09","Type":"ContainerDied","Data":"7f889bcadf7ce663727cbc8741c59aef00c48e3d003b6b6c48cbd3b09baf7a2a"}
Oct 30 13:27:19.491635 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:19.491040    5093 scope.go:122] "RemoveContainer" containerID="99894427068fa24ed1d6e3315a76a80a75c8e639b34e2861475299dff3781e42"
Oct 30 13:27:19.503826 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:19.503797    5093 scope.go:122] "RemoveContainer" containerID="99894427068fa24ed1d6e3315a76a80a75c8e639b34e2861475299dff3781e42"
Oct 30 13:27:19.504170 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:27:19.504145    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99894427068fa24ed1d6e3315a76a80a75c8e639b34e2861475299dff3781e42\": container with ID starting with 99894427068fa24ed1d6e3315a76a80a75c8e639b34e2861475299dff3781e42 not found: ID does not exist" containerID="99894427068fa24ed1d6e3315a76a80a75c8e639b34e2861475299dff3781e42"
Oct 30 13:27:19.504303 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:19.504176    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99894427068fa24ed1d6e3315a76a80a75c8e639b34e2861475299dff3781e42"} err="failed to get container status \"99894427068fa24ed1d6e3315a76a80a75c8e639b34e2861475299dff3781e42\": rpc error: code = NotFound desc = could not find container \"99894427068fa24ed1d6e3315a76a80a75c8e639b34e2861475299dff3781e42\": container with ID starting with 99894427068fa24ed1d6e3315a76a80a75c8e639b34e2861475299dff3781e42 not found: ID does not exist"
Oct 30 13:27:19.549361 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:19.549328    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-65e250-67df546b9d-xh95l"]
Oct 30 13:27:19.562006 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:19.561974    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-65e250-67df546b9d-xh95l"]
Oct 30 13:27:21.002415 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:21.002382    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09dab442-e495-4b11-91c4-97074cd89a09" path="/var/lib/kubelet/pods/09dab442-e495-4b11-91c4-97074cd89a09/volumes"
Oct 30 13:27:21.172963 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:27:21.172920    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:27:30.529156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:30.519330    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25" podUID="7805f89f-89d4-4754-b606-c3070f4df215" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 13:27:30.602641 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:30.601820    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg" podUID="ea079ac1-f502-4b5f-b16a-c91e2de98de8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Oct 30 13:27:30.610389 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:30.610240    5093 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25" podUID="7805f89f-89d4-4754-b606-c3070f4df215" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:27:30.639800 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:30.639283    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb" podUID="c4b4e00e-f58d-4a4e-b7db-19f724360db1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:27:30.650223 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:30.647240    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw" podUID="ad37954e-1350-4f75-a63f-a0c0f29ae416" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:27:30.957739 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:30.956991    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx" podUID="ba156eeb-9955-4f9a-857b-2c4d502a5fcc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:27:51.304753 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:27:51.304551    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:27:53.243166 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:53.236370    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt" podUID="4ae52c69-0c30-438f-9ac0-c87b6de16061" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:27:53.434156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:53.411245    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv" podUID="d6a44981-104f-4d3e-9937-d184a6ddc482" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:27:53.630212 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:53.608207    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg" podUID="ea079ac1-f502-4b5f-b16a-c91e2de98de8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Oct 30 13:27:53.631013 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:53.612583    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw" podUID="ad37954e-1350-4f75-a63f-a0c0f29ae416" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 13:27:53.635193 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:53.614342    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25" podUID="7805f89f-89d4-4754-b606-c3070f4df215" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 13:27:53.635193 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:27:53.614374    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb" podUID="c4b4e00e-f58d-4a4e-b7db-19f724360db1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Oct 30 13:28:01.326997 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:01.326959    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jdj4_f93ee017-ab79-447c-90e0-5b85b71dba24/console-operator/1.log"
Oct 30 13:28:01.328421 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:01.328397    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jdj4_f93ee017-ab79-447c-90e0-5b85b71dba24/console-operator/1.log"
Oct 30 13:28:01.332511 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:01.332490    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tnm_3a675ac7-fbe1-449b-accf-bb0456ed0bd3/ovn-acl-logging/0.log"
Oct 30 13:28:01.333878 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:01.333855    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tnm_3a675ac7-fbe1-449b-accf-bb0456ed0bd3/ovn-acl-logging/0.log"
Oct 30 13:28:15.284053 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:15.283248    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt" podUID="4ae52c69-0c30-438f-9ac0-c87b6de16061" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:28:15.426030 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:15.425969    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv" podUID="d6a44981-104f-4d3e-9937-d184a6ddc482" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:28:15.526114 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:15.526061    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25" podUID="7805f89f-89d4-4754-b606-c3070f4df215" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:28:15.526720 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:15.526691    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb" podUID="c4b4e00e-f58d-4a4e-b7db-19f724360db1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Oct 30 13:28:15.603963 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:15.603856    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw" podUID="ad37954e-1350-4f75-a63f-a0c0f29ae416" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:28:16.012156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:15.984320    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx" podUID="ba156eeb-9955-4f9a-857b-2c4d502a5fcc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:28:21.184448 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:28:21.184375    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:28:23.253856 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:23.253810    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7"]
Oct 30 13:28:23.254343 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:23.254090    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7" podUID="24a8f884-84d1-4852-80d1-fe699060a90a" containerName="istio-proxy" containerID="cri-o://12bcb869687f7f2f28e6c5f696a4a3da8e7df9a9b0988d6cb88aab99b5bd1f87" gracePeriod=30
Oct 30 13:28:26.519160 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:26.519089    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5cee02-7c59cdc9fb-6b47j"]
Oct 30 13:28:26.520348 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:26.520290    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5cee02-7c59cdc9fb-6b47j" podUID="ee1d1f98-eee4-47c2-8ef7-5204fa1d787f" containerName="maas-api" containerID="cri-o://8cde7a278617aebee75e41d40ee53029bb92647dcc4d6b262062217b8afaeded" gracePeriod=30
Oct 30 13:28:27.148832 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:27.148791    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5cee02-7c59cdc9fb-6b47j"
Oct 30 13:28:27.279286 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:27.279248    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee1d1f98-eee4-47c2-8ef7-5204fa1d787f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee1d1f98-eee4-47c2-8ef7-5204fa1d787f-maas-api-tls\") pod \"ee1d1f98-eee4-47c2-8ef7-5204fa1d787f\" (UID: \"ee1d1f98-eee4-47c2-8ef7-5204fa1d787f\") "
Oct 30 13:28:27.279485 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:27.279307    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee1d1f98-eee4-47c2-8ef7-5204fa1d787f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee1d1f98-eee4-47c2-8ef7-5204fa1d787f-openshift-service-ca\") pod \"ee1d1f98-eee4-47c2-8ef7-5204fa1d787f\" (UID: \"ee1d1f98-eee4-47c2-8ef7-5204fa1d787f\") "
Oct 30 13:28:27.279485 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:27.279346    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee1d1f98-eee4-47c2-8ef7-5204fa1d787f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee1d1f98-eee4-47c2-8ef7-5204fa1d787f-metrics-certs\") pod \"ee1d1f98-eee4-47c2-8ef7-5204fa1d787f\" (UID: \"ee1d1f98-eee4-47c2-8ef7-5204fa1d787f\") "
Oct 30 13:28:27.279619 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:27.279502    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee1d1f98-eee4-47c2-8ef7-5204fa1d787f-kube-api-access-gnppw\" (UniqueName: \"kubernetes.io/projected/ee1d1f98-eee4-47c2-8ef7-5204fa1d787f-kube-api-access-gnppw\") pod \"ee1d1f98-eee4-47c2-8ef7-5204fa1d787f\" (UID: \"ee1d1f98-eee4-47c2-8ef7-5204fa1d787f\") "
Oct 30 13:28:27.279772 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:27.279745    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee1d1f98-eee4-47c2-8ef7-5204fa1d787f-openshift-service-ca" pod "ee1d1f98-eee4-47c2-8ef7-5204fa1d787f" (UID: "ee1d1f98-eee4-47c2-8ef7-5204fa1d787f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:28:27.280313 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:27.280283    5093 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee1d1f98-eee4-47c2-8ef7-5204fa1d787f-openshift-service-ca\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:28:27.281604 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:27.281581    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee1d1f98-eee4-47c2-8ef7-5204fa1d787f-metrics-certs" pod "ee1d1f98-eee4-47c2-8ef7-5204fa1d787f" (UID: "ee1d1f98-eee4-47c2-8ef7-5204fa1d787f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:28:27.281700 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:27.281596    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee1d1f98-eee4-47c2-8ef7-5204fa1d787f-kube-api-access-gnppw" pod "ee1d1f98-eee4-47c2-8ef7-5204fa1d787f" (UID: "ee1d1f98-eee4-47c2-8ef7-5204fa1d787f"). InnerVolumeSpecName "kube-api-access-gnppw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:27.281700 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:27.281617    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee1d1f98-eee4-47c2-8ef7-5204fa1d787f-maas-api-tls" pod "ee1d1f98-eee4-47c2-8ef7-5204fa1d787f" (UID: "ee1d1f98-eee4-47c2-8ef7-5204fa1d787f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:28:27.383524 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:27.383420    5093 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee1d1f98-eee4-47c2-8ef7-5204fa1d787f-maas-api-tls\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:28:27.383524 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:27.383454    5093 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee1d1f98-eee4-47c2-8ef7-5204fa1d787f-metrics-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:28:27.383524 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:27.383467    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnppw\" (UniqueName: \"kubernetes.io/projected/ee1d1f98-eee4-47c2-8ef7-5204fa1d787f-kube-api-access-gnppw\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:28:27.624854 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:27.624818    5093 generic.go:356] "Generic (PLEG): container finished" podID="ee1d1f98-eee4-47c2-8ef7-5204fa1d787f" containerID="8cde7a278617aebee75e41d40ee53029bb92647dcc4d6b262062217b8afaeded" exitCode=0
Oct 30 13:28:27.625383 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:27.624876    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5cee02-7c59cdc9fb-6b47j"
Oct 30 13:28:27.625383 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:27.624899    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5cee02-7c59cdc9fb-6b47j" event={"ID":"ee1d1f98-eee4-47c2-8ef7-5204fa1d787f","Type":"ContainerDied","Data":"8cde7a278617aebee75e41d40ee53029bb92647dcc4d6b262062217b8afaeded"}
Oct 30 13:28:27.625383 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:27.624966    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5cee02-7c59cdc9fb-6b47j" event={"ID":"ee1d1f98-eee4-47c2-8ef7-5204fa1d787f","Type":"ContainerDied","Data":"e553136797760eb6ed398926ca741eb6c165a009f55e2f84c6dbf99557829952"}
Oct 30 13:28:27.625383 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:27.624987    5093 scope.go:122] "RemoveContainer" containerID="8cde7a278617aebee75e41d40ee53029bb92647dcc4d6b262062217b8afaeded"
Oct 30 13:28:27.636516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:27.636497    5093 scope.go:122] "RemoveContainer" containerID="8cde7a278617aebee75e41d40ee53029bb92647dcc4d6b262062217b8afaeded"
Oct 30 13:28:27.639502 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:28:27.639475    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cde7a278617aebee75e41d40ee53029bb92647dcc4d6b262062217b8afaeded\": container with ID starting with 8cde7a278617aebee75e41d40ee53029bb92647dcc4d6b262062217b8afaeded not found: ID does not exist" containerID="8cde7a278617aebee75e41d40ee53029bb92647dcc4d6b262062217b8afaeded"
Oct 30 13:28:27.639607 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:27.639516    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cde7a278617aebee75e41d40ee53029bb92647dcc4d6b262062217b8afaeded"} err="failed to get container status \"8cde7a278617aebee75e41d40ee53029bb92647dcc4d6b262062217b8afaeded\": rpc error: code = NotFound desc = could not find container \"8cde7a278617aebee75e41d40ee53029bb92647dcc4d6b262062217b8afaeded\": container with ID starting with 8cde7a278617aebee75e41d40ee53029bb92647dcc4d6b262062217b8afaeded not found: ID does not exist"
Oct 30 13:28:27.665846 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:27.665819    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h"]
Oct 30 13:28:27.666057 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:27.666029    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h" podUID="35df4dfa-f04b-47b8-86cc-8d7758fcec5a" containerName="istio-proxy" containerID="cri-o://23426448ba7bfae7d45b30d3947bddeb08864abdd57ca661250ab9d2179907d5" gracePeriod=30
Oct 30 13:28:27.709584 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:27.709543    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5cee02-7c59cdc9fb-6b47j"]
Oct 30 13:28:27.713237 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:27.713207    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5cee02-7c59cdc9fb-6b47j"]
Oct 30 13:28:28.544513 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:28.544441    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7"
Oct 30 13:28:28.637382 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:28.637345    5093 generic.go:356] "Generic (PLEG): container finished" podID="24a8f884-84d1-4852-80d1-fe699060a90a" containerID="12bcb869687f7f2f28e6c5f696a4a3da8e7df9a9b0988d6cb88aab99b5bd1f87" exitCode=0
Oct 30 13:28:28.637929 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:28.637402    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7"
Oct 30 13:28:28.638163 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:28.637426    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7" event={"ID":"24a8f884-84d1-4852-80d1-fe699060a90a","Type":"ContainerDied","Data":"12bcb869687f7f2f28e6c5f696a4a3da8e7df9a9b0988d6cb88aab99b5bd1f87"}
Oct 30 13:28:28.638288 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:28.638231    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7" event={"ID":"24a8f884-84d1-4852-80d1-fe699060a90a","Type":"ContainerDied","Data":"78e3554182cbb7a5f689e13628fd2eb3e746188592baf3731f31f042814f3f80"}
Oct 30 13:28:28.638288 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:28.638273    5093 scope.go:122] "RemoveContainer" containerID="12bcb869687f7f2f28e6c5f696a4a3da8e7df9a9b0988d6cb88aab99b5bd1f87"
Oct 30 13:28:28.652018 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:28.651994    5093 scope.go:122] "RemoveContainer" containerID="12bcb869687f7f2f28e6c5f696a4a3da8e7df9a9b0988d6cb88aab99b5bd1f87"
Oct 30 13:28:28.652634 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:28:28.652611    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12bcb869687f7f2f28e6c5f696a4a3da8e7df9a9b0988d6cb88aab99b5bd1f87\": container with ID starting with 12bcb869687f7f2f28e6c5f696a4a3da8e7df9a9b0988d6cb88aab99b5bd1f87 not found: ID does not exist" containerID="12bcb869687f7f2f28e6c5f696a4a3da8e7df9a9b0988d6cb88aab99b5bd1f87"
Oct 30 13:28:28.652742 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:28.652646    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12bcb869687f7f2f28e6c5f696a4a3da8e7df9a9b0988d6cb88aab99b5bd1f87"} err="failed to get container status \"12bcb869687f7f2f28e6c5f696a4a3da8e7df9a9b0988d6cb88aab99b5bd1f87\": rpc error: code = NotFound desc = could not find container \"12bcb869687f7f2f28e6c5f696a4a3da8e7df9a9b0988d6cb88aab99b5bd1f87\": container with ID starting with 12bcb869687f7f2f28e6c5f696a4a3da8e7df9a9b0988d6cb88aab99b5bd1f87 not found: ID does not exist"
Oct 30 13:28:28.702348 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:28.702280    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24a8f884-84d1-4852-80d1-fe699060a90a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24a8f884-84d1-4852-80d1-fe699060a90a-istiod-ca-cert\") pod \"24a8f884-84d1-4852-80d1-fe699060a90a\" (UID: \"24a8f884-84d1-4852-80d1-fe699060a90a\") "
Oct 30 13:28:28.702516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:28.702389    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24a8f884-84d1-4852-80d1-fe699060a90a-istio-token\" (UniqueName: \"kubernetes.io/projected/24a8f884-84d1-4852-80d1-fe699060a90a-istio-token\") pod \"24a8f884-84d1-4852-80d1-fe699060a90a\" (UID: \"24a8f884-84d1-4852-80d1-fe699060a90a\") "
Oct 30 13:28:28.702591 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:28.702516    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24a8f884-84d1-4852-80d1-fe699060a90a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24a8f884-84d1-4852-80d1-fe699060a90a-workload-certs\") pod \"24a8f884-84d1-4852-80d1-fe699060a90a\" (UID: \"24a8f884-84d1-4852-80d1-fe699060a90a\") "
Oct 30 13:28:28.702591 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:28.702560    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24a8f884-84d1-4852-80d1-fe699060a90a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a8f884-84d1-4852-80d1-fe699060a90a-credential-socket\") pod \"24a8f884-84d1-4852-80d1-fe699060a90a\" (UID: \"24a8f884-84d1-4852-80d1-fe699060a90a\") "
Oct 30 13:28:28.702714 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:28.702602    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24a8f884-84d1-4852-80d1-fe699060a90a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24a8f884-84d1-4852-80d1-fe699060a90a-istio-envoy\") pod \"24a8f884-84d1-4852-80d1-fe699060a90a\" (UID: \"24a8f884-84d1-4852-80d1-fe699060a90a\") "
Oct 30 13:28:28.702772 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:28.702760    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24a8f884-84d1-4852-80d1-fe699060a90a-kube-api-access-ktbpk\" (UniqueName: \"kubernetes.io/projected/24a8f884-84d1-4852-80d1-fe699060a90a-kube-api-access-ktbpk\") pod \"24a8f884-84d1-4852-80d1-fe699060a90a\" (UID: \"24a8f884-84d1-4852-80d1-fe699060a90a\") "
Oct 30 13:28:28.702836 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:28.702796    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24a8f884-84d1-4852-80d1-fe699060a90a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24a8f884-84d1-4852-80d1-fe699060a90a-istio-data\") pod \"24a8f884-84d1-4852-80d1-fe699060a90a\" (UID: \"24a8f884-84d1-4852-80d1-fe699060a90a\") "
Oct 30 13:28:28.702889 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:28.702836    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24a8f884-84d1-4852-80d1-fe699060a90a-workload-certs" pod "24a8f884-84d1-4852-80d1-fe699060a90a" (UID: "24a8f884-84d1-4852-80d1-fe699060a90a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:28:28.703097 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:28.703069    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24a8f884-84d1-4852-80d1-fe699060a90a-credential-socket" pod "24a8f884-84d1-4852-80d1-fe699060a90a" (UID: "24a8f884-84d1-4852-80d1-fe699060a90a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:28:28.703289 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:28.703226    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24a8f884-84d1-4852-80d1-fe699060a90a-istio-data" pod "24a8f884-84d1-4852-80d1-fe699060a90a" (UID: "24a8f884-84d1-4852-80d1-fe699060a90a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:28:28.703397 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:28.703373    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24a8f884-84d1-4852-80d1-fe699060a90a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24a8f884-84d1-4852-80d1-fe699060a90a-istio-podinfo\") pod \"24a8f884-84d1-4852-80d1-fe699060a90a\" (UID: \"24a8f884-84d1-4852-80d1-fe699060a90a\") "
Oct 30 13:28:28.703683 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:28.703373    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24a8f884-84d1-4852-80d1-fe699060a90a-istiod-ca-cert" pod "24a8f884-84d1-4852-80d1-fe699060a90a" (UID: "24a8f884-84d1-4852-80d1-fe699060a90a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:28:28.703807 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:28.703617    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24a8f884-84d1-4852-80d1-fe699060a90a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a8f884-84d1-4852-80d1-fe699060a90a-workload-socket\") pod \"24a8f884-84d1-4852-80d1-fe699060a90a\" (UID: \"24a8f884-84d1-4852-80d1-fe699060a90a\") "
Oct 30 13:28:28.703943 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:28.703918    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24a8f884-84d1-4852-80d1-fe699060a90a-workload-socket" pod "24a8f884-84d1-4852-80d1-fe699060a90a" (UID: "24a8f884-84d1-4852-80d1-fe699060a90a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:28:28.704614 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:28.704590    5093 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24a8f884-84d1-4852-80d1-fe699060a90a-workload-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:28:28.704751 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:28.704736    5093 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a8f884-84d1-4852-80d1-fe699060a90a-credential-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:28:28.704871 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:28.704835    5093 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24a8f884-84d1-4852-80d1-fe699060a90a-istio-data\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:28:28.705570 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:28.705538    5093 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a8f884-84d1-4852-80d1-fe699060a90a-workload-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:28:28.705570 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:28.705557    5093 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24a8f884-84d1-4852-80d1-fe699060a90a-istiod-ca-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:28:28.705570 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:28.705497    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24a8f884-84d1-4852-80d1-fe699060a90a-kube-api-access-ktbpk" pod "24a8f884-84d1-4852-80d1-fe699060a90a" (UID: "24a8f884-84d1-4852-80d1-fe699060a90a"). InnerVolumeSpecName "kube-api-access-ktbpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:28.705825 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:28.705632    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24a8f884-84d1-4852-80d1-fe699060a90a-istio-token" pod "24a8f884-84d1-4852-80d1-fe699060a90a" (UID: "24a8f884-84d1-4852-80d1-fe699060a90a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:28.705825 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:28.705734    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24a8f884-84d1-4852-80d1-fe699060a90a-istio-envoy" pod "24a8f884-84d1-4852-80d1-fe699060a90a" (UID: "24a8f884-84d1-4852-80d1-fe699060a90a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:28:28.705825 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:28.705800    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24a8f884-84d1-4852-80d1-fe699060a90a-istio-podinfo" pod "24a8f884-84d1-4852-80d1-fe699060a90a" (UID: "24a8f884-84d1-4852-80d1-fe699060a90a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:28:28.806764 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:28.806681    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktbpk\" (UniqueName: \"kubernetes.io/projected/24a8f884-84d1-4852-80d1-fe699060a90a-kube-api-access-ktbpk\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:28:28.806764 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:28.806714    5093 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24a8f884-84d1-4852-80d1-fe699060a90a-istio-podinfo\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:28:28.806764 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:28.806724    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24a8f884-84d1-4852-80d1-fe699060a90a-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:28:28.806764 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:28.806732    5093 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24a8f884-84d1-4852-80d1-fe699060a90a-istio-envoy\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:28:28.989837 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:28.989800    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7"]
Oct 30 13:28:29.014436 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:29.014398    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee1d1f98-eee4-47c2-8ef7-5204fa1d787f" path="/var/lib/kubelet/pods/ee1d1f98-eee4-47c2-8ef7-5204fa1d787f/volumes"
Oct 30 13:28:29.014882 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:29.014862    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-m4wz7"]
Oct 30 13:28:31.004434 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:31.004400    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24a8f884-84d1-4852-80d1-fe699060a90a" path="/var/lib/kubelet/pods/24a8f884-84d1-4852-80d1-fe699060a90a/volumes"
Oct 30 13:28:32.960984 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:32.960955    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h"
Oct 30 13:28:33.066649 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:33.066608    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-kube-api-access-zqrhl\" (UniqueName: \"kubernetes.io/projected/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-kube-api-access-zqrhl\") pod \"35df4dfa-f04b-47b8-86cc-8d7758fcec5a\" (UID: \"35df4dfa-f04b-47b8-86cc-8d7758fcec5a\") "
Oct 30 13:28:33.066905 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:33.066708    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-istio-podinfo\") pod \"35df4dfa-f04b-47b8-86cc-8d7758fcec5a\" (UID: \"35df4dfa-f04b-47b8-86cc-8d7758fcec5a\") "
Oct 30 13:28:33.066905 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:33.066747    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-istio-envoy\") pod \"35df4dfa-f04b-47b8-86cc-8d7758fcec5a\" (UID: \"35df4dfa-f04b-47b8-86cc-8d7758fcec5a\") "
Oct 30 13:28:33.066905 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:33.066775    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-credential-socket\") pod \"35df4dfa-f04b-47b8-86cc-8d7758fcec5a\" (UID: \"35df4dfa-f04b-47b8-86cc-8d7758fcec5a\") "
Oct 30 13:28:33.066905 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:33.066872    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-workload-socket\") pod \"35df4dfa-f04b-47b8-86cc-8d7758fcec5a\" (UID: \"35df4dfa-f04b-47b8-86cc-8d7758fcec5a\") "
Oct 30 13:28:33.068149 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:33.067582    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-istio-data" pod "35df4dfa-f04b-47b8-86cc-8d7758fcec5a" (UID: "35df4dfa-f04b-47b8-86cc-8d7758fcec5a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:28:33.068149 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:33.067040    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-istio-data\") pod \"35df4dfa-f04b-47b8-86cc-8d7758fcec5a\" (UID: \"35df4dfa-f04b-47b8-86cc-8d7758fcec5a\") "
Oct 30 13:28:33.069869 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:33.068589    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-workload-certs\") pod \"35df4dfa-f04b-47b8-86cc-8d7758fcec5a\" (UID: \"35df4dfa-f04b-47b8-86cc-8d7758fcec5a\") "
Oct 30 13:28:33.069869 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:33.068708    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-istiod-ca-cert\") pod \"35df4dfa-f04b-47b8-86cc-8d7758fcec5a\" (UID: \"35df4dfa-f04b-47b8-86cc-8d7758fcec5a\") "
Oct 30 13:28:33.069869 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:33.068776    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-istio-token\" (UniqueName: \"kubernetes.io/projected/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-istio-token\") pod \"35df4dfa-f04b-47b8-86cc-8d7758fcec5a\" (UID: \"35df4dfa-f04b-47b8-86cc-8d7758fcec5a\") "
Oct 30 13:28:33.069869 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:33.069422    5093 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-istio-data\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:28:33.087977 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:33.086671    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-workload-certs" pod "35df4dfa-f04b-47b8-86cc-8d7758fcec5a" (UID: "35df4dfa-f04b-47b8-86cc-8d7758fcec5a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:28:33.087977 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:33.086687    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-kube-api-access-zqrhl" pod "35df4dfa-f04b-47b8-86cc-8d7758fcec5a" (UID: "35df4dfa-f04b-47b8-86cc-8d7758fcec5a"). InnerVolumeSpecName "kube-api-access-zqrhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:33.087977 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:33.086929    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-credential-socket" pod "35df4dfa-f04b-47b8-86cc-8d7758fcec5a" (UID: "35df4dfa-f04b-47b8-86cc-8d7758fcec5a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:28:33.087977 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:33.087007    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-istiod-ca-cert" pod "35df4dfa-f04b-47b8-86cc-8d7758fcec5a" (UID: "35df4dfa-f04b-47b8-86cc-8d7758fcec5a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:28:33.087977 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:33.087273    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-workload-socket" pod "35df4dfa-f04b-47b8-86cc-8d7758fcec5a" (UID: "35df4dfa-f04b-47b8-86cc-8d7758fcec5a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:28:33.087977 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:33.087769    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-istio-envoy" pod "35df4dfa-f04b-47b8-86cc-8d7758fcec5a" (UID: "35df4dfa-f04b-47b8-86cc-8d7758fcec5a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:28:33.088449 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:33.088280    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-istio-token" pod "35df4dfa-f04b-47b8-86cc-8d7758fcec5a" (UID: "35df4dfa-f04b-47b8-86cc-8d7758fcec5a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:33.089445 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:33.089411    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-istio-podinfo" pod "35df4dfa-f04b-47b8-86cc-8d7758fcec5a" (UID: "35df4dfa-f04b-47b8-86cc-8d7758fcec5a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:28:33.170936 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:33.170905    5093 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-istiod-ca-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:28:33.170936 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:33.170933    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:28:33.170936 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:33.170943    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqrhl\" (UniqueName: \"kubernetes.io/projected/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-kube-api-access-zqrhl\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:28:33.171218 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:33.170953    5093 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-istio-podinfo\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:28:33.171218 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:33.170962    5093 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-istio-envoy\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:28:33.171218 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:33.170970    5093 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-credential-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:28:33.171218 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:33.170980    5093 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-workload-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:28:33.171218 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:33.170988    5093 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35df4dfa-f04b-47b8-86cc-8d7758fcec5a-workload-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:28:33.683146 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:33.683098    5093 generic.go:356] "Generic (PLEG): container finished" podID="35df4dfa-f04b-47b8-86cc-8d7758fcec5a" containerID="23426448ba7bfae7d45b30d3947bddeb08864abdd57ca661250ab9d2179907d5" exitCode=0
Oct 30 13:28:33.683330 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:33.683203    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h" event={"ID":"35df4dfa-f04b-47b8-86cc-8d7758fcec5a","Type":"ContainerDied","Data":"23426448ba7bfae7d45b30d3947bddeb08864abdd57ca661250ab9d2179907d5"}
Oct 30 13:28:33.683330 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:33.683228    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h"
Oct 30 13:28:33.683330 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:33.683248    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h" event={"ID":"35df4dfa-f04b-47b8-86cc-8d7758fcec5a","Type":"ContainerDied","Data":"c880e4c888d544dd457fe60c1f4310e304613d0cc62f834f9c1cef372b60574e"}
Oct 30 13:28:33.683330 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:33.683271    5093 scope.go:122] "RemoveContainer" containerID="23426448ba7bfae7d45b30d3947bddeb08864abdd57ca661250ab9d2179907d5"
Oct 30 13:28:33.695028 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:33.695007    5093 scope.go:122] "RemoveContainer" containerID="23426448ba7bfae7d45b30d3947bddeb08864abdd57ca661250ab9d2179907d5"
Oct 30 13:28:33.695371 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:28:33.695343    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23426448ba7bfae7d45b30d3947bddeb08864abdd57ca661250ab9d2179907d5\": container with ID starting with 23426448ba7bfae7d45b30d3947bddeb08864abdd57ca661250ab9d2179907d5 not found: ID does not exist" containerID="23426448ba7bfae7d45b30d3947bddeb08864abdd57ca661250ab9d2179907d5"
Oct 30 13:28:33.695455 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:33.695385    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23426448ba7bfae7d45b30d3947bddeb08864abdd57ca661250ab9d2179907d5"} err="failed to get container status \"23426448ba7bfae7d45b30d3947bddeb08864abdd57ca661250ab9d2179907d5\": rpc error: code = NotFound desc = could not find container \"23426448ba7bfae7d45b30d3947bddeb08864abdd57ca661250ab9d2179907d5\": container with ID starting with 23426448ba7bfae7d45b30d3947bddeb08864abdd57ca661250ab9d2179907d5 not found: ID does not exist"
Oct 30 13:28:33.738122 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:28:33.738076    5093 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35df4dfa_f04b_47b8_86cc_8d7758fcec5a.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35df4dfa_f04b_47b8_86cc_8d7758fcec5a.slice/crio-c880e4c888d544dd457fe60c1f4310e304613d0cc62f834f9c1cef372b60574e\": RecentStats: unable to find data in memory cache]"
Oct 30 13:28:33.753870 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:33.753837    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h"]
Oct 30 13:28:33.766488 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:33.766450    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5rw2h"]
Oct 30 13:28:34.999416 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:34.999378    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35df4dfa-f04b-47b8-86cc-8d7758fcec5a" path="/var/lib/kubelet/pods/35df4dfa-f04b-47b8-86cc-8d7758fcec5a/volumes"
Oct 30 13:28:51.167278 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:28:51.167239    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:28:59.528272 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:59.528220    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb" podUID="c4b4e00e-f58d-4a4e-b7db-19f724360db1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Oct 30 13:28:59.530121 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:59.530085    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25" podUID="7805f89f-89d4-4754-b606-c3070f4df215" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:28:59.619153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:28:59.615230    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw" podUID="ad37954e-1350-4f75-a63f-a0c0f29ae416" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:29:07.177720 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:07.177675    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c7vfw"]
Oct 30 13:29:07.180514 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:07.177933    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c7vfw" podUID="731d4cfd-f9fb-4f4c-bca1-4b632cd67156" containerName="discovery" containerID="cri-o://c7162acd6aac34553810cea785bcdf3e473101bf0e48bfc83470a5eef5ef5342" gracePeriod=30
Oct 30 13:29:07.491506 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:07.491478    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c7vfw"
Oct 30 13:29:07.600119 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:07.600082    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/731d4cfd-f9fb-4f4c-bca1-4b632cd67156-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/731d4cfd-f9fb-4f4c-bca1-4b632cd67156-istio-kubeconfig\") pod \"731d4cfd-f9fb-4f4c-bca1-4b632cd67156\" (UID: \"731d4cfd-f9fb-4f4c-bca1-4b632cd67156\") "
Oct 30 13:29:07.600338 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:07.600147    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/731d4cfd-f9fb-4f4c-bca1-4b632cd67156-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/731d4cfd-f9fb-4f4c-bca1-4b632cd67156-istio-csr-dns-cert\") pod \"731d4cfd-f9fb-4f4c-bca1-4b632cd67156\" (UID: \"731d4cfd-f9fb-4f4c-bca1-4b632cd67156\") "
Oct 30 13:29:07.600338 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:07.600216    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/731d4cfd-f9fb-4f4c-bca1-4b632cd67156-istio-token\" (UniqueName: \"kubernetes.io/projected/731d4cfd-f9fb-4f4c-bca1-4b632cd67156-istio-token\") pod \"731d4cfd-f9fb-4f4c-bca1-4b632cd67156\" (UID: \"731d4cfd-f9fb-4f4c-bca1-4b632cd67156\") "
Oct 30 13:29:07.600338 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:07.600279    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/731d4cfd-f9fb-4f4c-bca1-4b632cd67156-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/731d4cfd-f9fb-4f4c-bca1-4b632cd67156-istio-csr-ca-configmap\") pod \"731d4cfd-f9fb-4f4c-bca1-4b632cd67156\" (UID: \"731d4cfd-f9fb-4f4c-bca1-4b632cd67156\") "
Oct 30 13:29:07.600537 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:07.600351    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/731d4cfd-f9fb-4f4c-bca1-4b632cd67156-cacerts\" (UniqueName: \"kubernetes.io/secret/731d4cfd-f9fb-4f4c-bca1-4b632cd67156-cacerts\") pod \"731d4cfd-f9fb-4f4c-bca1-4b632cd67156\" (UID: \"731d4cfd-f9fb-4f4c-bca1-4b632cd67156\") "
Oct 30 13:29:07.600537 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:07.600415    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/731d4cfd-f9fb-4f4c-bca1-4b632cd67156-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/731d4cfd-f9fb-4f4c-bca1-4b632cd67156-local-certs\") pod \"731d4cfd-f9fb-4f4c-bca1-4b632cd67156\" (UID: \"731d4cfd-f9fb-4f4c-bca1-4b632cd67156\") "
Oct 30 13:29:07.600537 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:07.600521    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/731d4cfd-f9fb-4f4c-bca1-4b632cd67156-kube-api-access-s44xv\" (UniqueName: \"kubernetes.io/projected/731d4cfd-f9fb-4f4c-bca1-4b632cd67156-kube-api-access-s44xv\") pod \"731d4cfd-f9fb-4f4c-bca1-4b632cd67156\" (UID: \"731d4cfd-f9fb-4f4c-bca1-4b632cd67156\") "
Oct 30 13:29:07.602889 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:07.602854    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/731d4cfd-f9fb-4f4c-bca1-4b632cd67156-istio-csr-ca-configmap" pod "731d4cfd-f9fb-4f4c-bca1-4b632cd67156" (UID: "731d4cfd-f9fb-4f4c-bca1-4b632cd67156"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:29:07.603926 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:07.603897    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/731d4cfd-f9fb-4f4c-bca1-4b632cd67156-istio-kubeconfig" pod "731d4cfd-f9fb-4f4c-bca1-4b632cd67156" (UID: "731d4cfd-f9fb-4f4c-bca1-4b632cd67156"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:29:07.604272 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:07.604249    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/731d4cfd-f9fb-4f4c-bca1-4b632cd67156-istio-csr-dns-cert" pod "731d4cfd-f9fb-4f4c-bca1-4b632cd67156" (UID: "731d4cfd-f9fb-4f4c-bca1-4b632cd67156"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:29:07.604351 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:07.604325    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/731d4cfd-f9fb-4f4c-bca1-4b632cd67156-cacerts" pod "731d4cfd-f9fb-4f4c-bca1-4b632cd67156" (UID: "731d4cfd-f9fb-4f4c-bca1-4b632cd67156"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:29:07.604472 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:07.604454    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/731d4cfd-f9fb-4f4c-bca1-4b632cd67156-istio-token" pod "731d4cfd-f9fb-4f4c-bca1-4b632cd67156" (UID: "731d4cfd-f9fb-4f4c-bca1-4b632cd67156"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:29:07.604904 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:07.604874    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/731d4cfd-f9fb-4f4c-bca1-4b632cd67156-local-certs" pod "731d4cfd-f9fb-4f4c-bca1-4b632cd67156" (UID: "731d4cfd-f9fb-4f4c-bca1-4b632cd67156"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:29:07.604904 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:07.604893    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/731d4cfd-f9fb-4f4c-bca1-4b632cd67156-kube-api-access-s44xv" pod "731d4cfd-f9fb-4f4c-bca1-4b632cd67156" (UID: "731d4cfd-f9fb-4f4c-bca1-4b632cd67156"). InnerVolumeSpecName "kube-api-access-s44xv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:29:07.702318 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:07.702285    5093 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/731d4cfd-f9fb-4f4c-bca1-4b632cd67156-istio-kubeconfig\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:29:07.702318 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:07.702315    5093 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/731d4cfd-f9fb-4f4c-bca1-4b632cd67156-istio-csr-dns-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:29:07.702318 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:07.702325    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/731d4cfd-f9fb-4f4c-bca1-4b632cd67156-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:29:07.702572 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:07.702335    5093 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/731d4cfd-f9fb-4f4c-bca1-4b632cd67156-istio-csr-ca-configmap\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:29:07.702572 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:07.702345    5093 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/731d4cfd-f9fb-4f4c-bca1-4b632cd67156-cacerts\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:29:07.702572 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:07.702354    5093 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/731d4cfd-f9fb-4f4c-bca1-4b632cd67156-local-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:29:07.702572 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:07.702365    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s44xv\" (UniqueName: \"kubernetes.io/projected/731d4cfd-f9fb-4f4c-bca1-4b632cd67156-kube-api-access-s44xv\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:29:08.073966 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:08.073925    5093 generic.go:356] "Generic (PLEG): container finished" podID="731d4cfd-f9fb-4f4c-bca1-4b632cd67156" containerID="c7162acd6aac34553810cea785bcdf3e473101bf0e48bfc83470a5eef5ef5342" exitCode=0
Oct 30 13:29:08.074204 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:08.073992    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c7vfw"
Oct 30 13:29:08.074204 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:08.074089    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c7vfw" event={"ID":"731d4cfd-f9fb-4f4c-bca1-4b632cd67156","Type":"ContainerDied","Data":"c7162acd6aac34553810cea785bcdf3e473101bf0e48bfc83470a5eef5ef5342"}
Oct 30 13:29:08.074204 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:08.074155    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c7vfw" event={"ID":"731d4cfd-f9fb-4f4c-bca1-4b632cd67156","Type":"ContainerDied","Data":"39c1615f150e291b8e7c9359140780fe3483dcf53ea412dc8bb07fad105d1aa1"}
Oct 30 13:29:08.074204 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:08.074185    5093 scope.go:122] "RemoveContainer" containerID="c7162acd6aac34553810cea785bcdf3e473101bf0e48bfc83470a5eef5ef5342"
Oct 30 13:29:08.085696 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:08.085679    5093 scope.go:122] "RemoveContainer" containerID="c7162acd6aac34553810cea785bcdf3e473101bf0e48bfc83470a5eef5ef5342"
Oct 30 13:29:08.085980 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:29:08.085955    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7162acd6aac34553810cea785bcdf3e473101bf0e48bfc83470a5eef5ef5342\": container with ID starting with c7162acd6aac34553810cea785bcdf3e473101bf0e48bfc83470a5eef5ef5342 not found: ID does not exist" containerID="c7162acd6aac34553810cea785bcdf3e473101bf0e48bfc83470a5eef5ef5342"
Oct 30 13:29:08.086045 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:08.085994    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7162acd6aac34553810cea785bcdf3e473101bf0e48bfc83470a5eef5ef5342"} err="failed to get container status \"c7162acd6aac34553810cea785bcdf3e473101bf0e48bfc83470a5eef5ef5342\": rpc error: code = NotFound desc = could not find container \"c7162acd6aac34553810cea785bcdf3e473101bf0e48bfc83470a5eef5ef5342\": container with ID starting with c7162acd6aac34553810cea785bcdf3e473101bf0e48bfc83470a5eef5ef5342 not found: ID does not exist"
Oct 30 13:29:08.131386 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:08.131347    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c7vfw"]
Oct 30 13:29:08.142378 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:08.142347    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c7vfw"]
Oct 30 13:29:09.000589 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:09.000550    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="731d4cfd-f9fb-4f4c-bca1-4b632cd67156" path="/var/lib/kubelet/pods/731d4cfd-f9fb-4f4c-bca1-4b632cd67156/volumes"
Oct 30 13:29:10.508839 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:10.508799    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b"]
Oct 30 13:29:10.509360 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:10.509036    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b" podUID="9829402b-5c78-4bef-bff1-bb1faf135ce6" containerName="istio-proxy" containerID="cri-o://4713c1b37b81d8fada8a07282c052cfce3434eabcabcdbf04c7849befa5cbba1" gracePeriod=30
Oct 30 13:29:15.800578 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:15.800546    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b"
Oct 30 13:29:15.918485 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:15.918449    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9829402b-5c78-4bef-bff1-bb1faf135ce6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9829402b-5c78-4bef-bff1-bb1faf135ce6-istio-podinfo\") pod \"9829402b-5c78-4bef-bff1-bb1faf135ce6\" (UID: \"9829402b-5c78-4bef-bff1-bb1faf135ce6\") "
Oct 30 13:29:15.918653 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:15.918537    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9829402b-5c78-4bef-bff1-bb1faf135ce6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9829402b-5c78-4bef-bff1-bb1faf135ce6-istiod-ca-cert\") pod \"9829402b-5c78-4bef-bff1-bb1faf135ce6\" (UID: \"9829402b-5c78-4bef-bff1-bb1faf135ce6\") "
Oct 30 13:29:15.918653 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:15.918599    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9829402b-5c78-4bef-bff1-bb1faf135ce6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9829402b-5c78-4bef-bff1-bb1faf135ce6-istio-data\") pod \"9829402b-5c78-4bef-bff1-bb1faf135ce6\" (UID: \"9829402b-5c78-4bef-bff1-bb1faf135ce6\") "
Oct 30 13:29:15.918738 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:15.918669    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9829402b-5c78-4bef-bff1-bb1faf135ce6-istio-token\" (UniqueName: \"kubernetes.io/projected/9829402b-5c78-4bef-bff1-bb1faf135ce6-istio-token\") pod \"9829402b-5c78-4bef-bff1-bb1faf135ce6\" (UID: \"9829402b-5c78-4bef-bff1-bb1faf135ce6\") "
Oct 30 13:29:15.918738 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:15.918692    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9829402b-5c78-4bef-bff1-bb1faf135ce6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9829402b-5c78-4bef-bff1-bb1faf135ce6-workload-socket\") pod \"9829402b-5c78-4bef-bff1-bb1faf135ce6\" (UID: \"9829402b-5c78-4bef-bff1-bb1faf135ce6\") "
Oct 30 13:29:15.918833 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:15.918761    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9829402b-5c78-4bef-bff1-bb1faf135ce6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9829402b-5c78-4bef-bff1-bb1faf135ce6-istio-envoy\") pod \"9829402b-5c78-4bef-bff1-bb1faf135ce6\" (UID: \"9829402b-5c78-4bef-bff1-bb1faf135ce6\") "
Oct 30 13:29:15.918833 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:15.918794    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9829402b-5c78-4bef-bff1-bb1faf135ce6-kube-api-access-r7txh\" (UniqueName: \"kubernetes.io/projected/9829402b-5c78-4bef-bff1-bb1faf135ce6-kube-api-access-r7txh\") pod \"9829402b-5c78-4bef-bff1-bb1faf135ce6\" (UID: \"9829402b-5c78-4bef-bff1-bb1faf135ce6\") "
Oct 30 13:29:15.918941 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:15.918842    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9829402b-5c78-4bef-bff1-bb1faf135ce6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9829402b-5c78-4bef-bff1-bb1faf135ce6-workload-certs\") pod \"9829402b-5c78-4bef-bff1-bb1faf135ce6\" (UID: \"9829402b-5c78-4bef-bff1-bb1faf135ce6\") "
Oct 30 13:29:15.918941 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:15.918901    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9829402b-5c78-4bef-bff1-bb1faf135ce6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9829402b-5c78-4bef-bff1-bb1faf135ce6-credential-socket\") pod \"9829402b-5c78-4bef-bff1-bb1faf135ce6\" (UID: \"9829402b-5c78-4bef-bff1-bb1faf135ce6\") "
Oct 30 13:29:15.919053 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:15.918932    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9829402b-5c78-4bef-bff1-bb1faf135ce6-istio-data" pod "9829402b-5c78-4bef-bff1-bb1faf135ce6" (UID: "9829402b-5c78-4bef-bff1-bb1faf135ce6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:29:15.919053 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:15.919038    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9829402b-5c78-4bef-bff1-bb1faf135ce6-istiod-ca-cert" pod "9829402b-5c78-4bef-bff1-bb1faf135ce6" (UID: "9829402b-5c78-4bef-bff1-bb1faf135ce6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:29:15.919160 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:15.919041    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9829402b-5c78-4bef-bff1-bb1faf135ce6-workload-socket" pod "9829402b-5c78-4bef-bff1-bb1faf135ce6" (UID: "9829402b-5c78-4bef-bff1-bb1faf135ce6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:29:15.919247 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:15.919211    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9829402b-5c78-4bef-bff1-bb1faf135ce6-workload-certs" pod "9829402b-5c78-4bef-bff1-bb1faf135ce6" (UID: "9829402b-5c78-4bef-bff1-bb1faf135ce6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:29:15.919374 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:15.919354    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9829402b-5c78-4bef-bff1-bb1faf135ce6-credential-socket" pod "9829402b-5c78-4bef-bff1-bb1faf135ce6" (UID: "9829402b-5c78-4bef-bff1-bb1faf135ce6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:29:15.919722 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:15.919697    5093 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9829402b-5c78-4bef-bff1-bb1faf135ce6-istiod-ca-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:29:15.919827 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:15.919728    5093 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9829402b-5c78-4bef-bff1-bb1faf135ce6-istio-data\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:29:15.919827 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:15.919743    5093 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9829402b-5c78-4bef-bff1-bb1faf135ce6-workload-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:29:15.919827 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:15.919757    5093 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9829402b-5c78-4bef-bff1-bb1faf135ce6-workload-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:29:15.919827 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:15.919772    5093 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9829402b-5c78-4bef-bff1-bb1faf135ce6-credential-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:29:15.920884 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:15.920862    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9829402b-5c78-4bef-bff1-bb1faf135ce6-istio-podinfo" pod "9829402b-5c78-4bef-bff1-bb1faf135ce6" (UID: "9829402b-5c78-4bef-bff1-bb1faf135ce6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:29:15.921076 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:15.921053    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9829402b-5c78-4bef-bff1-bb1faf135ce6-kube-api-access-r7txh" pod "9829402b-5c78-4bef-bff1-bb1faf135ce6" (UID: "9829402b-5c78-4bef-bff1-bb1faf135ce6"). InnerVolumeSpecName "kube-api-access-r7txh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:29:15.921265 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:15.921081    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9829402b-5c78-4bef-bff1-bb1faf135ce6-istio-token" pod "9829402b-5c78-4bef-bff1-bb1faf135ce6" (UID: "9829402b-5c78-4bef-bff1-bb1faf135ce6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:29:15.921476 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:15.921284    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9829402b-5c78-4bef-bff1-bb1faf135ce6-istio-envoy" pod "9829402b-5c78-4bef-bff1-bb1faf135ce6" (UID: "9829402b-5c78-4bef-bff1-bb1faf135ce6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:29:16.020886 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.020844    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9829402b-5c78-4bef-bff1-bb1faf135ce6-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:29:16.020886 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.020877    5093 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9829402b-5c78-4bef-bff1-bb1faf135ce6-istio-envoy\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:29:16.020886 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.020891    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7txh\" (UniqueName: \"kubernetes.io/projected/9829402b-5c78-4bef-bff1-bb1faf135ce6-kube-api-access-r7txh\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:29:16.021155 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.020904    5093 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9829402b-5c78-4bef-bff1-bb1faf135ce6-istio-podinfo\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:29:16.135249 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.135124    5093 generic.go:356] "Generic (PLEG): container finished" podID="9829402b-5c78-4bef-bff1-bb1faf135ce6" containerID="4713c1b37b81d8fada8a07282c052cfce3434eabcabcdbf04c7849befa5cbba1" exitCode=0
Oct 30 13:29:16.135249 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.135217    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b"
Oct 30 13:29:16.135560 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.135255    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b" event={"ID":"9829402b-5c78-4bef-bff1-bb1faf135ce6","Type":"ContainerDied","Data":"4713c1b37b81d8fada8a07282c052cfce3434eabcabcdbf04c7849befa5cbba1"}
Oct 30 13:29:16.135560 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.135327    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b" event={"ID":"9829402b-5c78-4bef-bff1-bb1faf135ce6","Type":"ContainerDied","Data":"2e93670e4749c3790d881722631ca74f6332bb82fc5edbe10557ca5dd4b9ca18"}
Oct 30 13:29:16.135560 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.135358    5093 scope.go:122] "RemoveContainer" containerID="4713c1b37b81d8fada8a07282c052cfce3434eabcabcdbf04c7849befa5cbba1"
Oct 30 13:29:16.149038 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.149010    5093 scope.go:122] "RemoveContainer" containerID="4713c1b37b81d8fada8a07282c052cfce3434eabcabcdbf04c7849befa5cbba1"
Oct 30 13:29:16.149388 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:29:16.149364    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4713c1b37b81d8fada8a07282c052cfce3434eabcabcdbf04c7849befa5cbba1\": container with ID starting with 4713c1b37b81d8fada8a07282c052cfce3434eabcabcdbf04c7849befa5cbba1 not found: ID does not exist" containerID="4713c1b37b81d8fada8a07282c052cfce3434eabcabcdbf04c7849befa5cbba1"
Oct 30 13:29:16.149468 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.149397    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4713c1b37b81d8fada8a07282c052cfce3434eabcabcdbf04c7849befa5cbba1"} err="failed to get container status \"4713c1b37b81d8fada8a07282c052cfce3434eabcabcdbf04c7849befa5cbba1\": rpc error: code = NotFound desc = could not find container \"4713c1b37b81d8fada8a07282c052cfce3434eabcabcdbf04c7849befa5cbba1\": container with ID starting with 4713c1b37b81d8fada8a07282c052cfce3434eabcabcdbf04c7849befa5cbba1 not found: ID does not exist"
Oct 30 13:29:16.193371 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.193338    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b"]
Oct 30 13:29:16.196727 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.196703    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-w5l8b"]
Oct 30 13:29:16.457983 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.457946    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x"]
Oct 30 13:29:16.458838 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.458813    5093 cpu_manager.go:414] "containerMap: removing container" podUID="731d4cfd-f9fb-4f4c-bca1-4b632cd67156" containerName="discovery"
Oct 30 13:29:16.458838 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.458835    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="731d4cfd-f9fb-4f4c-bca1-4b632cd67156" containerName="discovery"
Oct 30 13:29:16.458987 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.458853    5093 cpu_manager.go:414] "containerMap: removing container" podUID="09dab442-e495-4b11-91c4-97074cd89a09" containerName="payload-processing"
Oct 30 13:29:16.458987 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.458860    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09dab442-e495-4b11-91c4-97074cd89a09" containerName="payload-processing"
Oct 30 13:29:16.458987 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.458912    5093 cpu_manager.go:414] "containerMap: removing container" podUID="ee1d1f98-eee4-47c2-8ef7-5204fa1d787f" containerName="maas-api"
Oct 30 13:29:16.458987 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.458923    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee1d1f98-eee4-47c2-8ef7-5204fa1d787f" containerName="maas-api"
Oct 30 13:29:16.458987 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.458936    5093 cpu_manager.go:414] "containerMap: removing container" podUID="35df4dfa-f04b-47b8-86cc-8d7758fcec5a" containerName="istio-proxy"
Oct 30 13:29:16.458987 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.458944    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35df4dfa-f04b-47b8-86cc-8d7758fcec5a" containerName="istio-proxy"
Oct 30 13:29:16.458987 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.458960    5093 cpu_manager.go:414] "containerMap: removing container" podUID="24a8f884-84d1-4852-80d1-fe699060a90a" containerName="istio-proxy"
Oct 30 13:29:16.458987 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.458967    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24a8f884-84d1-4852-80d1-fe699060a90a" containerName="istio-proxy"
Oct 30 13:29:16.458987 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.458988    5093 cpu_manager.go:414] "containerMap: removing container" podUID="9829402b-5c78-4bef-bff1-bb1faf135ce6" containerName="istio-proxy"
Oct 30 13:29:16.459362 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.458995    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9829402b-5c78-4bef-bff1-bb1faf135ce6" containerName="istio-proxy"
Oct 30 13:29:16.459362 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.459113    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee1d1f98-eee4-47c2-8ef7-5204fa1d787f" containerName="maas-api"
Oct 30 13:29:16.459362 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.459123    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="35df4dfa-f04b-47b8-86cc-8d7758fcec5a" containerName="istio-proxy"
Oct 30 13:29:16.459362 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.459158    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="24a8f884-84d1-4852-80d1-fe699060a90a" containerName="istio-proxy"
Oct 30 13:29:16.459362 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.459170    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="9829402b-5c78-4bef-bff1-bb1faf135ce6" containerName="istio-proxy"
Oct 30 13:29:16.459362 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.459184    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="731d4cfd-f9fb-4f4c-bca1-4b632cd67156" containerName="discovery"
Oct 30 13:29:16.459362 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.459193    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="09dab442-e495-4b11-91c4-97074cd89a09" containerName="payload-processing"
Oct 30 13:29:16.464660 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.464635    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x"
Oct 30 13:29:16.485594 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.484887    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-167965-openshift-default-dockercfg-rfp6j\""
Oct 30 13:29:16.494454 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.487294    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x"]
Oct 30 13:29:16.634498 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.633929    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-credential-socket\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x\" (UID: \"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x"
Oct 30 13:29:16.634498 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.634008    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-istio-token\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x\" (UID: \"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x"
Oct 30 13:29:16.634498 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.634057    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-workload-socket\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x\" (UID: \"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x"
Oct 30 13:29:16.634498 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.634093    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-workload-certs\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x\" (UID: \"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x"
Oct 30 13:29:16.634498 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.634198    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-istio-podinfo\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x\" (UID: \"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x"
Oct 30 13:29:16.634498 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.634240    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzv4f\" (UniqueName: \"kubernetes.io/projected/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-kube-api-access-jzv4f\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x\" (UID: \"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x"
Oct 30 13:29:16.634498 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.634314    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-istiod-ca-cert\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x\" (UID: \"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x"
Oct 30 13:29:16.634498 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.634428    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-istio-envoy\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x\" (UID: \"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x"
Oct 30 13:29:16.634498 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.634467    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-istio-data\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x\" (UID: \"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x"
Oct 30 13:29:16.736194 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.736094    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzv4f\" (UniqueName: \"kubernetes.io/projected/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-kube-api-access-jzv4f\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x\" (UID: \"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x"
Oct 30 13:29:16.736367 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.736195    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-istiod-ca-cert\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x\" (UID: \"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x"
Oct 30 13:29:16.736367 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.736267    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-istio-envoy\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x\" (UID: \"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x"
Oct 30 13:29:16.736367 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.736299    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-istio-data\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x\" (UID: \"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x"
Oct 30 13:29:16.736546 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.736396    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-credential-socket\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x\" (UID: \"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x"
Oct 30 13:29:16.736546 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.736451    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-istio-token\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x\" (UID: \"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x"
Oct 30 13:29:16.736546 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.736489    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-workload-socket\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x\" (UID: \"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x"
Oct 30 13:29:16.736546 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.736522    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-workload-certs\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x\" (UID: \"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x"
Oct 30 13:29:16.736777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.736616    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-istio-podinfo\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x\" (UID: \"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x"
Oct 30 13:29:16.736841 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.736807    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-credential-socket\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x\" (UID: \"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x"
Oct 30 13:29:16.737035 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.737004    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-workload-socket\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x\" (UID: \"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x"
Oct 30 13:29:16.737151 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.737054    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-workload-certs\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x\" (UID: \"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x"
Oct 30 13:29:16.737151 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.737095    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-istiod-ca-cert\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x\" (UID: \"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x"
Oct 30 13:29:16.737338 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.737311    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-istio-data\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x\" (UID: \"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x"
Oct 30 13:29:16.739451 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.739425    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-istio-podinfo\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x\" (UID: \"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x"
Oct 30 13:29:16.743408 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.743375    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-istio-envoy\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x\" (UID: \"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x"
Oct 30 13:29:16.772160 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.758179    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-istio-token\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x\" (UID: \"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x"
Oct 30 13:29:16.772160 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.760397    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzv4f\" (UniqueName: \"kubernetes.io/projected/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-kube-api-access-jzv4f\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x\" (UID: \"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x"
Oct 30 13:29:16.804211 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:16.802816    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x"
Oct 30 13:29:17.006358 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:17.006259    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9829402b-5c78-4bef-bff1-bb1faf135ce6" path="/var/lib/kubelet/pods/9829402b-5c78-4bef-bff1-bb1faf135ce6/volumes"
Oct 30 13:29:17.133366 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:17.133330    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x"]
Oct 30 13:29:17.133519 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:29:17.133493    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a9a0d53_3fb0_414a_a543_f5dd4ed39eef.slice/crio-a557d49240ef6dd7446c711b8d909c36223a48397f601e9a9cb7f94574908c43 WatchSource:0}: Error finding container a557d49240ef6dd7446c711b8d909c36223a48397f601e9a9cb7f94574908c43: Status 404 returned error can't find the container with id a557d49240ef6dd7446c711b8d909c36223a48397f601e9a9cb7f94574908c43
Oct 30 13:29:17.136275 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:17.136236    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:29:17.136372 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:17.136305    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:29:17.136372 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:17.136334    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:29:17.147332 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:17.147298    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x" event={"ID":"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef","Type":"ContainerStarted","Data":"a557d49240ef6dd7446c711b8d909c36223a48397f601e9a9cb7f94574908c43"}
Oct 30 13:29:17.452302 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:17.452265    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c684d2-568d5d66c9-xlfrb"]
Oct 30 13:29:17.456748 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:17.456724    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c684d2-568d5d66c9-xlfrb"
Oct 30 13:29:17.473589 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:17.473547    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-c684d2-serving-cert\""
Oct 30 13:29:17.477185 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:17.477094    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c684d2-568d5d66c9-xlfrb"]
Oct 30 13:29:17.551181 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:17.551142    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec7e602c-fb95-46e6-bbaa-74dd09dbebe5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c684d2-568d5d66c9-xlfrb\" (UID: \"ec7e602c-fb95-46e6-bbaa-74dd09dbebe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c684d2-568d5d66c9-xlfrb"
Oct 30 13:29:17.551386 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:17.551232    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9krc5\" (UniqueName: \"kubernetes.io/projected/ec7e602c-fb95-46e6-bbaa-74dd09dbebe5-kube-api-access-9krc5\") pod \"maas-api-e2e-shared-a-w2-c684d2-568d5d66c9-xlfrb\" (UID: \"ec7e602c-fb95-46e6-bbaa-74dd09dbebe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c684d2-568d5d66c9-xlfrb"
Oct 30 13:29:17.551386 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:17.551286    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec7e602c-fb95-46e6-bbaa-74dd09dbebe5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-c684d2-568d5d66c9-xlfrb\" (UID: \"ec7e602c-fb95-46e6-bbaa-74dd09dbebe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c684d2-568d5d66c9-xlfrb"
Oct 30 13:29:17.551505 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:17.551444    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec7e602c-fb95-46e6-bbaa-74dd09dbebe5-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-c684d2-568d5d66c9-xlfrb\" (UID: \"ec7e602c-fb95-46e6-bbaa-74dd09dbebe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c684d2-568d5d66c9-xlfrb"
Oct 30 13:29:17.665267 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:17.653232    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9krc5\" (UniqueName: \"kubernetes.io/projected/ec7e602c-fb95-46e6-bbaa-74dd09dbebe5-kube-api-access-9krc5\") pod \"maas-api-e2e-shared-a-w2-c684d2-568d5d66c9-xlfrb\" (UID: \"ec7e602c-fb95-46e6-bbaa-74dd09dbebe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c684d2-568d5d66c9-xlfrb"
Oct 30 13:29:17.665267 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:17.653312    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec7e602c-fb95-46e6-bbaa-74dd09dbebe5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-c684d2-568d5d66c9-xlfrb\" (UID: \"ec7e602c-fb95-46e6-bbaa-74dd09dbebe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c684d2-568d5d66c9-xlfrb"
Oct 30 13:29:17.665267 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:17.653507    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec7e602c-fb95-46e6-bbaa-74dd09dbebe5-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-c684d2-568d5d66c9-xlfrb\" (UID: \"ec7e602c-fb95-46e6-bbaa-74dd09dbebe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c684d2-568d5d66c9-xlfrb"
Oct 30 13:29:17.665267 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:17.653645    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec7e602c-fb95-46e6-bbaa-74dd09dbebe5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c684d2-568d5d66c9-xlfrb\" (UID: \"ec7e602c-fb95-46e6-bbaa-74dd09dbebe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c684d2-568d5d66c9-xlfrb"
Oct 30 13:29:17.665267 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:17.665185    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec7e602c-fb95-46e6-bbaa-74dd09dbebe5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-c684d2-568d5d66c9-xlfrb\" (UID: \"ec7e602c-fb95-46e6-bbaa-74dd09dbebe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c684d2-568d5d66c9-xlfrb"
Oct 30 13:29:17.675556 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:17.675473    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec7e602c-fb95-46e6-bbaa-74dd09dbebe5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c684d2-568d5d66c9-xlfrb\" (UID: \"ec7e602c-fb95-46e6-bbaa-74dd09dbebe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c684d2-568d5d66c9-xlfrb"
Oct 30 13:29:17.679353 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:17.679321    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9krc5\" (UniqueName: \"kubernetes.io/projected/ec7e602c-fb95-46e6-bbaa-74dd09dbebe5-kube-api-access-9krc5\") pod \"maas-api-e2e-shared-a-w2-c684d2-568d5d66c9-xlfrb\" (UID: \"ec7e602c-fb95-46e6-bbaa-74dd09dbebe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c684d2-568d5d66c9-xlfrb"
Oct 30 13:29:17.681013 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:17.680982    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec7e602c-fb95-46e6-bbaa-74dd09dbebe5-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-c684d2-568d5d66c9-xlfrb\" (UID: \"ec7e602c-fb95-46e6-bbaa-74dd09dbebe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c684d2-568d5d66c9-xlfrb"
Oct 30 13:29:17.775022 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:17.774930    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c684d2-568d5d66c9-xlfrb"
Oct 30 13:29:18.083583 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:18.083549    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c684d2-568d5d66c9-xlfrb"]
Oct 30 13:29:18.085337 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:29:18.085293    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec7e602c_fb95_46e6_bbaa_74dd09dbebe5.slice/crio-e7eb7a43a2ae6390df1d1e1b6b28289656d99e3bcb9b73eca3e4275ddebaac90 WatchSource:0}: Error finding container e7eb7a43a2ae6390df1d1e1b6b28289656d99e3bcb9b73eca3e4275ddebaac90: Status 404 returned error can't find the container with id e7eb7a43a2ae6390df1d1e1b6b28289656d99e3bcb9b73eca3e4275ddebaac90
Oct 30 13:29:18.157406 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:18.157367    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c684d2-568d5d66c9-xlfrb" event={"ID":"ec7e602c-fb95-46e6-bbaa-74dd09dbebe5","Type":"ContainerStarted","Data":"e7eb7a43a2ae6390df1d1e1b6b28289656d99e3bcb9b73eca3e4275ddebaac90"}
Oct 30 13:29:18.159392 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:18.159360    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x" event={"ID":"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef","Type":"ContainerStarted","Data":"e6d77701831809fd79f318075f38754db29a881178681c1154bc41ee91ddc86f"}
Oct 30 13:29:18.207677 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:18.207603    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x" podStartSLOduration=2.207578587 podStartE2EDuration="2.207578587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:18.204482523 +0000 UTC m=+1577.351749762" watchObservedRunningTime="2026-10-30 13:29:18.207578587 +0000 UTC m=+1577.354845829"
Oct 30 13:29:18.803343 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:18.803265    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x"
Oct 30 13:29:18.809024 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:18.808996    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x"
Oct 30 13:29:19.176596 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:19.176103    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c684d2-568d5d66c9-xlfrb" event={"ID":"ec7e602c-fb95-46e6-bbaa-74dd09dbebe5","Type":"ContainerStarted","Data":"82107c5a156eefa9712b1fdf147b43055a74a167627431e00cfe6fdb8e198939"}
Oct 30 13:29:19.176596 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:19.176215    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x"
Oct 30 13:29:19.176596 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:19.176262    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c684d2-568d5d66c9-xlfrb"
Oct 30 13:29:19.177336 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:19.177304    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x"
Oct 30 13:29:19.232168 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:19.231086    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c684d2-568d5d66c9-xlfrb" podStartSLOduration=1.8747450350000001 podStartE2EDuration="2.231067895s" totalImagesPullingTime="356.32286ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:19.230472913 +0000 UTC m=+1578.377740154" watchObservedRunningTime="2026-10-30 13:29:19.231067895 +0000 UTC m=+1578.378335139"
Oct 30 13:29:20.586157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:20.583256    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25" podUID="7805f89f-89d4-4754-b606-c3070f4df215" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:29:20.620202 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:20.612160    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg" podUID="ea079ac1-f502-4b5f-b16a-c91e2de98de8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:29:20.687157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:20.683619    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp"]
Oct 30 13:29:20.911050 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:20.877733    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp"
Oct 30 13:29:20.939548 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:20.939506    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-cab6c5-openshift-default-dockercfg-dsbh2\""
Oct 30 13:29:21.061262 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:21.061216    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx" podUID="ba156eeb-9955-4f9a-857b-2c4d502a5fcc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 30 13:29:21.144158 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:21.066900    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc75f0e6-64c6-4fde-9476-3859350da986-istio-envoy\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp\" (UID: \"cc75f0e6-64c6-4fde-9476-3859350da986\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp"
Oct 30 13:29:21.144158 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:21.066964    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc75f0e6-64c6-4fde-9476-3859350da986-istiod-ca-cert\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp\" (UID: \"cc75f0e6-64c6-4fde-9476-3859350da986\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp"
Oct 30 13:29:21.144158 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:21.067010    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc75f0e6-64c6-4fde-9476-3859350da986-workload-socket\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp\" (UID: \"cc75f0e6-64c6-4fde-9476-3859350da986\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp"
Oct 30 13:29:21.144158 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:21.067047    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc75f0e6-64c6-4fde-9476-3859350da986-credential-socket\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp\" (UID: \"cc75f0e6-64c6-4fde-9476-3859350da986\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp"
Oct 30 13:29:21.144158 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:21.067076    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc75f0e6-64c6-4fde-9476-3859350da986-workload-certs\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp\" (UID: \"cc75f0e6-64c6-4fde-9476-3859350da986\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp"
Oct 30 13:29:21.144158 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:21.067117    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc75f0e6-64c6-4fde-9476-3859350da986-istio-podinfo\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp\" (UID: \"cc75f0e6-64c6-4fde-9476-3859350da986\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp"
Oct 30 13:29:21.144158 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:21.067218    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc75f0e6-64c6-4fde-9476-3859350da986-istio-token\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp\" (UID: \"cc75f0e6-64c6-4fde-9476-3859350da986\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp"
Oct 30 13:29:21.144158 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:21.067293    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc75f0e6-64c6-4fde-9476-3859350da986-istio-data\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp\" (UID: \"cc75f0e6-64c6-4fde-9476-3859350da986\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp"
Oct 30 13:29:21.144158 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:21.067331    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2d6t\" (UniqueName: \"kubernetes.io/projected/cc75f0e6-64c6-4fde-9476-3859350da986-kube-api-access-k2d6t\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp\" (UID: \"cc75f0e6-64c6-4fde-9476-3859350da986\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp"
Oct 30 13:29:21.196569 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:21.172638    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc75f0e6-64c6-4fde-9476-3859350da986-istio-token\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp\" (UID: \"cc75f0e6-64c6-4fde-9476-3859350da986\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp"
Oct 30 13:29:21.196569 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:21.172750    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc75f0e6-64c6-4fde-9476-3859350da986-istio-data\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp\" (UID: \"cc75f0e6-64c6-4fde-9476-3859350da986\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp"
Oct 30 13:29:21.196569 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:21.172786    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2d6t\" (UniqueName: \"kubernetes.io/projected/cc75f0e6-64c6-4fde-9476-3859350da986-kube-api-access-k2d6t\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp\" (UID: \"cc75f0e6-64c6-4fde-9476-3859350da986\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp"
Oct 30 13:29:21.196569 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:21.172897    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc75f0e6-64c6-4fde-9476-3859350da986-istio-envoy\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp\" (UID: \"cc75f0e6-64c6-4fde-9476-3859350da986\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp"
Oct 30 13:29:21.196569 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:21.172944    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc75f0e6-64c6-4fde-9476-3859350da986-istiod-ca-cert\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp\" (UID: \"cc75f0e6-64c6-4fde-9476-3859350da986\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp"
Oct 30 13:29:21.196569 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:21.172995    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc75f0e6-64c6-4fde-9476-3859350da986-workload-socket\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp\" (UID: \"cc75f0e6-64c6-4fde-9476-3859350da986\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp"
Oct 30 13:29:21.196569 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:21.173037    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc75f0e6-64c6-4fde-9476-3859350da986-credential-socket\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp\" (UID: \"cc75f0e6-64c6-4fde-9476-3859350da986\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp"
Oct 30 13:29:21.196569 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:21.173074    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc75f0e6-64c6-4fde-9476-3859350da986-workload-certs\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp\" (UID: \"cc75f0e6-64c6-4fde-9476-3859350da986\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp"
Oct 30 13:29:21.196569 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:21.173123    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc75f0e6-64c6-4fde-9476-3859350da986-istio-podinfo\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp\" (UID: \"cc75f0e6-64c6-4fde-9476-3859350da986\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp"
Oct 30 13:29:21.196569 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:21.187367    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc75f0e6-64c6-4fde-9476-3859350da986-istio-data\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp\" (UID: \"cc75f0e6-64c6-4fde-9476-3859350da986\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp"
Oct 30 13:29:21.223108 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:21.197867    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc75f0e6-64c6-4fde-9476-3859350da986-istiod-ca-cert\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp\" (UID: \"cc75f0e6-64c6-4fde-9476-3859350da986\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp"
Oct 30 13:29:21.223108 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:21.202737    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc75f0e6-64c6-4fde-9476-3859350da986-credential-socket\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp\" (UID: \"cc75f0e6-64c6-4fde-9476-3859350da986\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp"
Oct 30 13:29:21.223108 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:21.222787    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc75f0e6-64c6-4fde-9476-3859350da986-workload-certs\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp\" (UID: \"cc75f0e6-64c6-4fde-9476-3859350da986\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp"
Oct 30 13:29:21.223108 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:21.223059    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc75f0e6-64c6-4fde-9476-3859350da986-workload-socket\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp\" (UID: \"cc75f0e6-64c6-4fde-9476-3859350da986\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp"
Oct 30 13:29:21.246513 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:21.246463    5093 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt" podUID="4ae52c69-0c30-438f-9ac0-c87b6de16061" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:29:21.325844 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:21.319915    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc75f0e6-64c6-4fde-9476-3859350da986-istio-envoy\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp\" (UID: \"cc75f0e6-64c6-4fde-9476-3859350da986\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp"
Oct 30 13:29:21.325844 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:21.322418    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc75f0e6-64c6-4fde-9476-3859350da986-istio-token\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp\" (UID: \"cc75f0e6-64c6-4fde-9476-3859350da986\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp"
Oct 30 13:29:21.325844 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:29:21.324518    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:29:21.372593 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:21.372556    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp"]
Oct 30 13:29:21.385975 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:21.385908    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2d6t\" (UniqueName: \"kubernetes.io/projected/cc75f0e6-64c6-4fde-9476-3859350da986-kube-api-access-k2d6t\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp\" (UID: \"cc75f0e6-64c6-4fde-9476-3859350da986\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp"
Oct 30 13:29:21.416176 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:21.413468    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc75f0e6-64c6-4fde-9476-3859350da986-istio-podinfo\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp\" (UID: \"cc75f0e6-64c6-4fde-9476-3859350da986\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp"
Oct 30 13:29:21.510681 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:21.510393    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp"
Oct 30 13:29:21.868446 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:29:21.868414    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc75f0e6_64c6_4fde_9476_3859350da986.slice/crio-0bfd8d4b3904165b722b499849aada63d32fe050f73b08dca945ab022560cbc5 WatchSource:0}: Error finding container 0bfd8d4b3904165b722b499849aada63d32fe050f73b08dca945ab022560cbc5: Status 404 returned error can't find the container with id 0bfd8d4b3904165b722b499849aada63d32fe050f73b08dca945ab022560cbc5
Oct 30 13:29:21.869273 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:21.869243    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp"]
Oct 30 13:29:21.871697 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:21.871653    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:29:21.871781 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:21.871762    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:29:21.871821 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:21.871806    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:29:22.391148 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:22.391093    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp" event={"ID":"cc75f0e6-64c6-4fde-9476-3859350da986","Type":"ContainerStarted","Data":"b2c240320d8110c9d359ce33e13d8f8d1fde4db9e02e3c491dab4ca21fff450b"}
Oct 30 13:29:22.391353 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:22.391180    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp" event={"ID":"cc75f0e6-64c6-4fde-9476-3859350da986","Type":"ContainerStarted","Data":"0bfd8d4b3904165b722b499849aada63d32fe050f73b08dca945ab022560cbc5"}
Oct 30 13:29:22.450950 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:22.450882    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp" podStartSLOduration=2.450862245 podStartE2EDuration="2.450862245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:22.44464569 +0000 UTC m=+1581.591912929" watchObservedRunningTime="2026-10-30 13:29:22.450862245 +0000 UTC m=+1581.598129487"
Oct 30 13:29:22.511838 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:22.511794    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp"
Oct 30 13:29:22.517202 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:22.517175    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp"
Oct 30 13:29:23.216829 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:23.216105    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-167965-6b479587d8-zktwq"]
Oct 30 13:29:23.221569 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:23.221543    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-167965-6b479587d8-zktwq"
Oct 30 13:29:23.242521 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:23.242489    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-167965-6b479587d8-zktwq"]
Oct 30 13:29:23.249325 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:23.249295    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-167965\""
Oct 30 13:29:23.249498 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:23.249404    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-167965-dockercfg-2phtq\""
Oct 30 13:29:23.263245 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:23.263206    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njzz8\" (UniqueName: \"kubernetes.io/projected/1f39b9a2-0b8a-4eb7-9e7d-9a90d357672c-kube-api-access-njzz8\") pod \"payload-processing-e2e-shared-a-w4-167965-6b479587d8-zktwq\" (UID: \"1f39b9a2-0b8a-4eb7-9e7d-9a90d357672c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-167965-6b479587d8-zktwq"
Oct 30 13:29:23.263539 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:23.263516    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f39b9a2-0b8a-4eb7-9e7d-9a90d357672c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-167965-6b479587d8-zktwq\" (UID: \"1f39b9a2-0b8a-4eb7-9e7d-9a90d357672c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-167965-6b479587d8-zktwq"
Oct 30 13:29:23.279603 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:23.279568    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-167965-fdb89d64c-td5pc"]
Oct 30 13:29:23.287906 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:23.287878    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-167965-fdb89d64c-td5pc"
Oct 30 13:29:23.300667 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:23.300638    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-167965-fdb89d64c-td5pc"]
Oct 30 13:29:23.413211 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:23.411513    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f39b9a2-0b8a-4eb7-9e7d-9a90d357672c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-167965-6b479587d8-zktwq\" (UID: \"1f39b9a2-0b8a-4eb7-9e7d-9a90d357672c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-167965-6b479587d8-zktwq"
Oct 30 13:29:23.413211 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:23.411633    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njzz8\" (UniqueName: \"kubernetes.io/projected/1f39b9a2-0b8a-4eb7-9e7d-9a90d357672c-kube-api-access-njzz8\") pod \"payload-processing-e2e-shared-a-w4-167965-6b479587d8-zktwq\" (UID: \"1f39b9a2-0b8a-4eb7-9e7d-9a90d357672c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-167965-6b479587d8-zktwq"
Oct 30 13:29:23.414227 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:23.414195    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f39b9a2-0b8a-4eb7-9e7d-9a90d357672c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-167965-6b479587d8-zktwq\" (UID: \"1f39b9a2-0b8a-4eb7-9e7d-9a90d357672c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-167965-6b479587d8-zktwq"
Oct 30 13:29:23.455318 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:23.455288    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp"
Oct 30 13:29:23.456973 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:23.456950    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp"
Oct 30 13:29:23.458583 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:23.458555    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njzz8\" (UniqueName: \"kubernetes.io/projected/1f39b9a2-0b8a-4eb7-9e7d-9a90d357672c-kube-api-access-njzz8\") pod \"payload-processing-e2e-shared-a-w4-167965-6b479587d8-zktwq\" (UID: \"1f39b9a2-0b8a-4eb7-9e7d-9a90d357672c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-167965-6b479587d8-zktwq"
Oct 30 13:29:23.512621 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:23.512517    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27a0b223-6a47-4bf9-8f6e-dd5a8bc80912-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-167965-fdb89d64c-td5pc\" (UID: \"27a0b223-6a47-4bf9-8f6e-dd5a8bc80912\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-167965-fdb89d64c-td5pc"
Oct 30 13:29:23.512939 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:23.512910    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jf2v\" (UniqueName: \"kubernetes.io/projected/27a0b223-6a47-4bf9-8f6e-dd5a8bc80912-kube-api-access-7jf2v\") pod \"payload-pre-processing-e2e-shared-a-w4-167965-fdb89d64c-td5pc\" (UID: \"27a0b223-6a47-4bf9-8f6e-dd5a8bc80912\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-167965-fdb89d64c-td5pc"
Oct 30 13:29:23.542427 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:23.542392    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-167965-6b479587d8-zktwq"
Oct 30 13:29:23.613885 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:23.613854    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jf2v\" (UniqueName: \"kubernetes.io/projected/27a0b223-6a47-4bf9-8f6e-dd5a8bc80912-kube-api-access-7jf2v\") pod \"payload-pre-processing-e2e-shared-a-w4-167965-fdb89d64c-td5pc\" (UID: \"27a0b223-6a47-4bf9-8f6e-dd5a8bc80912\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-167965-fdb89d64c-td5pc"
Oct 30 13:29:23.614031 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:23.613918    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27a0b223-6a47-4bf9-8f6e-dd5a8bc80912-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-167965-fdb89d64c-td5pc\" (UID: \"27a0b223-6a47-4bf9-8f6e-dd5a8bc80912\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-167965-fdb89d64c-td5pc"
Oct 30 13:29:23.619475 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:23.619451    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27a0b223-6a47-4bf9-8f6e-dd5a8bc80912-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-167965-fdb89d64c-td5pc\" (UID: \"27a0b223-6a47-4bf9-8f6e-dd5a8bc80912\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-167965-fdb89d64c-td5pc"
Oct 30 13:29:23.659733 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:23.651723    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jf2v\" (UniqueName: \"kubernetes.io/projected/27a0b223-6a47-4bf9-8f6e-dd5a8bc80912-kube-api-access-7jf2v\") pod \"payload-pre-processing-e2e-shared-a-w4-167965-fdb89d64c-td5pc\" (UID: \"27a0b223-6a47-4bf9-8f6e-dd5a8bc80912\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-167965-fdb89d64c-td5pc"
Oct 30 13:29:23.863814 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:23.863777    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-167965-6b479587d8-zktwq"]
Oct 30 13:29:23.864665 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:29:23.864632    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f39b9a2_0b8a_4eb7_9e7d_9a90d357672c.slice/crio-1e51eabfef65de87f4e697a52b30995162ff67b1cfdf5877e90c6d0e583d6b11 WatchSource:0}: Error finding container 1e51eabfef65de87f4e697a52b30995162ff67b1cfdf5877e90c6d0e583d6b11: Status 404 returned error can't find the container with id 1e51eabfef65de87f4e697a52b30995162ff67b1cfdf5877e90c6d0e583d6b11
Oct 30 13:29:23.905043 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:23.905013    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-167965-fdb89d64c-td5pc"
Oct 30 13:29:24.221480 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:24.221443    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-167965-fdb89d64c-td5pc"]
Oct 30 13:29:24.222413 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:29:24.222379    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27a0b223_6a47_4bf9_8f6e_dd5a8bc80912.slice/crio-a9afdf01cefad36492f495bfd32267fcb0376a6b9ca0f1edab449e2dd60e163e WatchSource:0}: Error finding container a9afdf01cefad36492f495bfd32267fcb0376a6b9ca0f1edab449e2dd60e163e: Status 404 returned error can't find the container with id a9afdf01cefad36492f495bfd32267fcb0376a6b9ca0f1edab449e2dd60e163e
Oct 30 13:29:24.465966 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:24.465925    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-167965-6b479587d8-zktwq" event={"ID":"1f39b9a2-0b8a-4eb7-9e7d-9a90d357672c","Type":"ContainerStarted","Data":"659d5097d58c262f7a47798d916ca7756cd69256ba8fcb213cb859a3efd6b19c"}
Oct 30 13:29:24.466169 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:24.465994    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-167965-6b479587d8-zktwq" event={"ID":"1f39b9a2-0b8a-4eb7-9e7d-9a90d357672c","Type":"ContainerStarted","Data":"1e51eabfef65de87f4e697a52b30995162ff67b1cfdf5877e90c6d0e583d6b11"}
Oct 30 13:29:24.466362 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:24.466333    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-167965-6b479587d8-zktwq"
Oct 30 13:29:24.468836 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:24.468157    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-167965-fdb89d64c-td5pc" event={"ID":"27a0b223-6a47-4bf9-8f6e-dd5a8bc80912","Type":"ContainerStarted","Data":"a93d067852c3eccf13e72988bbb4992964af2be0f59297e5fa88dafcf7305a49"}
Oct 30 13:29:24.468836 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:24.468382    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-167965-fdb89d64c-td5pc"
Oct 30 13:29:24.468836 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:24.468417    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-167965-fdb89d64c-td5pc" event={"ID":"27a0b223-6a47-4bf9-8f6e-dd5a8bc80912","Type":"ContainerStarted","Data":"a9afdf01cefad36492f495bfd32267fcb0376a6b9ca0f1edab449e2dd60e163e"}
Oct 30 13:29:24.509555 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:24.509455    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-167965-6b479587d8-zktwq" podStartSLOduration=1.5094401689999999 podStartE2EDuration="1.509440169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:24.505659125 +0000 UTC m=+1583.652926365" watchObservedRunningTime="2026-10-30 13:29:24.509440169 +0000 UTC m=+1583.656707409"
Oct 30 13:29:24.548825 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:24.548756    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-167965-fdb89d64c-td5pc" podStartSLOduration=1.548736543 podStartE2EDuration="1.548736543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:24.546409227 +0000 UTC m=+1583.693676468" watchObservedRunningTime="2026-10-30 13:29:24.548736543 +0000 UTC m=+1583.696003787"
Oct 30 13:29:26.387603 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:26.387571    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c684d2-568d5d66c9-xlfrb"
Oct 30 13:29:26.647502 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:26.647406    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-65e250-58c5679c7-lbn2k"]
Oct 30 13:29:26.647677 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:26.647654    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-65e250-58c5679c7-lbn2k" podUID="6b4e2373-e7d6-451d-9213-257c6b56ef16" containerName="maas-api" containerID="cri-o://3d0548fb72044a63d6587fa6c21562a6b4dd5a4d574e8e7b9dece7142509240a" gracePeriod=30
Oct 30 13:29:26.831232 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:26.831187    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cab6c5-866cf94959-h48bz"]
Oct 30 13:29:26.836750 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:26.836720    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cab6c5-866cf94959-h48bz"
Oct 30 13:29:26.865655 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:26.865541    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-cab6c5\""
Oct 30 13:29:26.866012 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:26.865974    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70b59b98-9888-4d17-8695-e16a21f08fb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-cab6c5-866cf94959-h48bz\" (UID: \"70b59b98-9888-4d17-8695-e16a21f08fb0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cab6c5-866cf94959-h48bz"
Oct 30 13:29:26.866125 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:26.866059    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnfcc\" (UniqueName: \"kubernetes.io/projected/70b59b98-9888-4d17-8695-e16a21f08fb0-kube-api-access-wnfcc\") pod \"payload-pre-processing-e2e-shared-a-w3-cab6c5-866cf94959-h48bz\" (UID: \"70b59b98-9888-4d17-8695-e16a21f08fb0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cab6c5-866cf94959-h48bz"
Oct 30 13:29:26.867280 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:26.867254    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-cab6c5-555d87bc58-6slkr"]
Oct 30 13:29:26.867587 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:26.867549    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-cab6c5-dockercfg-c4t6m\""
Oct 30 13:29:26.873904 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:26.873878    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cab6c5-555d87bc58-6slkr"
Oct 30 13:29:26.874890 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:26.874865    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cab6c5-866cf94959-h48bz"]
Oct 30 13:29:26.899586 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:26.899498    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-cab6c5-555d87bc58-6slkr"]
Oct 30 13:29:26.974339 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:26.971078    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70b59b98-9888-4d17-8695-e16a21f08fb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-cab6c5-866cf94959-h48bz\" (UID: \"70b59b98-9888-4d17-8695-e16a21f08fb0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cab6c5-866cf94959-h48bz"
Oct 30 13:29:26.974339 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:26.971162    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnfcc\" (UniqueName: \"kubernetes.io/projected/70b59b98-9888-4d17-8695-e16a21f08fb0-kube-api-access-wnfcc\") pod \"payload-pre-processing-e2e-shared-a-w3-cab6c5-866cf94959-h48bz\" (UID: \"70b59b98-9888-4d17-8695-e16a21f08fb0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cab6c5-866cf94959-h48bz"
Oct 30 13:29:26.974339 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:26.971274    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bcb5af7-5a52-48b1-88fa-e5b1359baced-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-cab6c5-555d87bc58-6slkr\" (UID: \"6bcb5af7-5a52-48b1-88fa-e5b1359baced\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cab6c5-555d87bc58-6slkr"
Oct 30 13:29:26.974339 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:26.971483    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl2pg\" (UniqueName: \"kubernetes.io/projected/6bcb5af7-5a52-48b1-88fa-e5b1359baced-kube-api-access-fl2pg\") pod \"payload-processing-e2e-shared-a-w3-cab6c5-555d87bc58-6slkr\" (UID: \"6bcb5af7-5a52-48b1-88fa-e5b1359baced\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cab6c5-555d87bc58-6slkr"
Oct 30 13:29:26.974339 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:26.972534    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70b59b98-9888-4d17-8695-e16a21f08fb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-cab6c5-866cf94959-h48bz\" (UID: \"70b59b98-9888-4d17-8695-e16a21f08fb0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cab6c5-866cf94959-h48bz"
Oct 30 13:29:27.000390 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:26.994700    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnfcc\" (UniqueName: \"kubernetes.io/projected/70b59b98-9888-4d17-8695-e16a21f08fb0-kube-api-access-wnfcc\") pod \"payload-pre-processing-e2e-shared-a-w3-cab6c5-866cf94959-h48bz\" (UID: \"70b59b98-9888-4d17-8695-e16a21f08fb0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cab6c5-866cf94959-h48bz"
Oct 30 13:29:27.006248 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:27.006108    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-65e250-58c5679c7-lbn2k"
Oct 30 13:29:27.072947 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:27.072916    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b4e2373-e7d6-451d-9213-257c6b56ef16-kube-api-access-p2z2b\" (UniqueName: \"kubernetes.io/projected/6b4e2373-e7d6-451d-9213-257c6b56ef16-kube-api-access-p2z2b\") pod \"6b4e2373-e7d6-451d-9213-257c6b56ef16\" (UID: \"6b4e2373-e7d6-451d-9213-257c6b56ef16\") "
Oct 30 13:29:27.073180 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:27.072961    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b4e2373-e7d6-451d-9213-257c6b56ef16-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b4e2373-e7d6-451d-9213-257c6b56ef16-metrics-certs\") pod \"6b4e2373-e7d6-451d-9213-257c6b56ef16\" (UID: \"6b4e2373-e7d6-451d-9213-257c6b56ef16\") "
Oct 30 13:29:27.073180 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:27.072997    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b4e2373-e7d6-451d-9213-257c6b56ef16-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b4e2373-e7d6-451d-9213-257c6b56ef16-openshift-service-ca\") pod \"6b4e2373-e7d6-451d-9213-257c6b56ef16\" (UID: \"6b4e2373-e7d6-451d-9213-257c6b56ef16\") "
Oct 30 13:29:27.073180 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:27.073050    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b4e2373-e7d6-451d-9213-257c6b56ef16-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b4e2373-e7d6-451d-9213-257c6b56ef16-maas-api-tls\") pod \"6b4e2373-e7d6-451d-9213-257c6b56ef16\" (UID: \"6b4e2373-e7d6-451d-9213-257c6b56ef16\") "
Oct 30 13:29:27.073473 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:27.073451    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bcb5af7-5a52-48b1-88fa-e5b1359baced-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-cab6c5-555d87bc58-6slkr\" (UID: \"6bcb5af7-5a52-48b1-88fa-e5b1359baced\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cab6c5-555d87bc58-6slkr"
Oct 30 13:29:27.073543 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:27.073461    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b4e2373-e7d6-451d-9213-257c6b56ef16-openshift-service-ca" pod "6b4e2373-e7d6-451d-9213-257c6b56ef16" (UID: "6b4e2373-e7d6-451d-9213-257c6b56ef16"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:29:27.073789 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:27.073760    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl2pg\" (UniqueName: \"kubernetes.io/projected/6bcb5af7-5a52-48b1-88fa-e5b1359baced-kube-api-access-fl2pg\") pod \"payload-processing-e2e-shared-a-w3-cab6c5-555d87bc58-6slkr\" (UID: \"6bcb5af7-5a52-48b1-88fa-e5b1359baced\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cab6c5-555d87bc58-6slkr"
Oct 30 13:29:27.074086 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:27.074062    5093 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b4e2373-e7d6-451d-9213-257c6b56ef16-openshift-service-ca\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:29:27.074205 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:27.074170    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bcb5af7-5a52-48b1-88fa-e5b1359baced-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-cab6c5-555d87bc58-6slkr\" (UID: \"6bcb5af7-5a52-48b1-88fa-e5b1359baced\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cab6c5-555d87bc58-6slkr"
Oct 30 13:29:27.076943 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:27.076916    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b4e2373-e7d6-451d-9213-257c6b56ef16-metrics-certs" pod "6b4e2373-e7d6-451d-9213-257c6b56ef16" (UID: "6b4e2373-e7d6-451d-9213-257c6b56ef16"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:29:27.076943 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:27.076927    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b4e2373-e7d6-451d-9213-257c6b56ef16-kube-api-access-p2z2b" pod "6b4e2373-e7d6-451d-9213-257c6b56ef16" (UID: "6b4e2373-e7d6-451d-9213-257c6b56ef16"). InnerVolumeSpecName "kube-api-access-p2z2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:29:27.077655 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:27.077628    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b4e2373-e7d6-451d-9213-257c6b56ef16-maas-api-tls" pod "6b4e2373-e7d6-451d-9213-257c6b56ef16" (UID: "6b4e2373-e7d6-451d-9213-257c6b56ef16"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:29:27.096197 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:27.096121    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl2pg\" (UniqueName: \"kubernetes.io/projected/6bcb5af7-5a52-48b1-88fa-e5b1359baced-kube-api-access-fl2pg\") pod \"payload-processing-e2e-shared-a-w3-cab6c5-555d87bc58-6slkr\" (UID: \"6bcb5af7-5a52-48b1-88fa-e5b1359baced\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cab6c5-555d87bc58-6slkr"
Oct 30 13:29:27.175327 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:27.175291    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2z2b\" (UniqueName: \"kubernetes.io/projected/6b4e2373-e7d6-451d-9213-257c6b56ef16-kube-api-access-p2z2b\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:29:27.175327 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:27.175327    5093 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b4e2373-e7d6-451d-9213-257c6b56ef16-metrics-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:29:27.175546 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:27.175340    5093 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b4e2373-e7d6-451d-9213-257c6b56ef16-maas-api-tls\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:29:27.185283 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:27.185249    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cab6c5-866cf94959-h48bz"
Oct 30 13:29:27.198790 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:27.198763    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cab6c5-555d87bc58-6slkr"
Oct 30 13:29:27.505588 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:27.505487    5093 generic.go:356] "Generic (PLEG): container finished" podID="6b4e2373-e7d6-451d-9213-257c6b56ef16" containerID="3d0548fb72044a63d6587fa6c21562a6b4dd5a4d574e8e7b9dece7142509240a" exitCode=0
Oct 30 13:29:27.505588 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:27.505543    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-65e250-58c5679c7-lbn2k"
Oct 30 13:29:27.506164 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:27.505574    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-65e250-58c5679c7-lbn2k" event={"ID":"6b4e2373-e7d6-451d-9213-257c6b56ef16","Type":"ContainerDied","Data":"3d0548fb72044a63d6587fa6c21562a6b4dd5a4d574e8e7b9dece7142509240a"}
Oct 30 13:29:27.506164 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:27.505639    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-65e250-58c5679c7-lbn2k" event={"ID":"6b4e2373-e7d6-451d-9213-257c6b56ef16","Type":"ContainerDied","Data":"cf0c6ff89a5f32c33056d17b7ba8acc16efc5f4e1ddebbc5866a0fc55ac1fef0"}
Oct 30 13:29:27.506164 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:27.505672    5093 scope.go:122] "RemoveContainer" containerID="3d0548fb72044a63d6587fa6c21562a6b4dd5a4d574e8e7b9dece7142509240a"
Oct 30 13:29:27.530228 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:27.530202    5093 scope.go:122] "RemoveContainer" containerID="3d0548fb72044a63d6587fa6c21562a6b4dd5a4d574e8e7b9dece7142509240a"
Oct 30 13:29:27.530583 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:29:27.530557    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d0548fb72044a63d6587fa6c21562a6b4dd5a4d574e8e7b9dece7142509240a\": container with ID starting with 3d0548fb72044a63d6587fa6c21562a6b4dd5a4d574e8e7b9dece7142509240a not found: ID does not exist" containerID="3d0548fb72044a63d6587fa6c21562a6b4dd5a4d574e8e7b9dece7142509240a"
Oct 30 13:29:27.530701 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:27.530591    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d0548fb72044a63d6587fa6c21562a6b4dd5a4d574e8e7b9dece7142509240a"} err="failed to get container status \"3d0548fb72044a63d6587fa6c21562a6b4dd5a4d574e8e7b9dece7142509240a\": rpc error: code = NotFound desc = could not find container \"3d0548fb72044a63d6587fa6c21562a6b4dd5a4d574e8e7b9dece7142509240a\": container with ID starting with 3d0548fb72044a63d6587fa6c21562a6b4dd5a4d574e8e7b9dece7142509240a not found: ID does not exist"
Oct 30 13:29:27.543457 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:27.543428    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cab6c5-866cf94959-h48bz"]
Oct 30 13:29:27.563729 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:27.563698    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-65e250-58c5679c7-lbn2k"]
Oct 30 13:29:27.573652 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:27.573622    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-65e250-58c5679c7-lbn2k"]
Oct 30 13:29:27.593490 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:27.593460    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-cab6c5-555d87bc58-6slkr"]
Oct 30 13:29:27.608147 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:29:27.605270    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bcb5af7_5a52_48b1_88fa_e5b1359baced.slice/crio-9b891c71837a2f528aad02919b8e27c31a3cc98b1f3ccb102d99de969a0c7d3f WatchSource:0}: Error finding container 9b891c71837a2f528aad02919b8e27c31a3cc98b1f3ccb102d99de969a0c7d3f: Status 404 returned error can't find the container with id 9b891c71837a2f528aad02919b8e27c31a3cc98b1f3ccb102d99de969a0c7d3f
Oct 30 13:29:28.520165 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:28.515786    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cab6c5-555d87bc58-6slkr" event={"ID":"6bcb5af7-5a52-48b1-88fa-e5b1359baced","Type":"ContainerStarted","Data":"e8049095d31a5da478724d2e649f76e7d86f1da5f89705b70b2d03807df994ff"}
Oct 30 13:29:28.520165 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:28.515856    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cab6c5-555d87bc58-6slkr"
Oct 30 13:29:28.520165 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:28.515884    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cab6c5-555d87bc58-6slkr" event={"ID":"6bcb5af7-5a52-48b1-88fa-e5b1359baced","Type":"ContainerStarted","Data":"9b891c71837a2f528aad02919b8e27c31a3cc98b1f3ccb102d99de969a0c7d3f"}
Oct 30 13:29:28.520165 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:28.518807    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cab6c5-866cf94959-h48bz" event={"ID":"70b59b98-9888-4d17-8695-e16a21f08fb0","Type":"ContainerStarted","Data":"86322fd73affd84416a817e2af7955e48a56583913a34f94c5f4ea82bee54714"}
Oct 30 13:29:28.520165 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:28.518849    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cab6c5-866cf94959-h48bz" event={"ID":"70b59b98-9888-4d17-8695-e16a21f08fb0","Type":"ContainerStarted","Data":"b6350e9ee987d2cb4460ef5f4c196fe95bc4e0bd488043ed4b21d54d9d3bd3c5"}
Oct 30 13:29:28.520165 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:28.518949    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cab6c5-866cf94959-h48bz"
Oct 30 13:29:28.561261 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:28.561199    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cab6c5-555d87bc58-6slkr" podStartSLOduration=2.561186135 podStartE2EDuration="2.561186135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:28.557003754 +0000 UTC m=+1587.704270996" watchObservedRunningTime="2026-10-30 13:29:28.561186135 +0000 UTC m=+1587.708453376"
Oct 30 13:29:28.605221 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:28.605154    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cab6c5-866cf94959-h48bz" podStartSLOduration=2.605122171 podStartE2EDuration="2.605122171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:28.597840872 +0000 UTC m=+1587.745108112" watchObservedRunningTime="2026-10-30 13:29:28.605122171 +0000 UTC m=+1587.752389413"
Oct 30 13:29:29.014211 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:29.014176    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b4e2373-e7d6-451d-9213-257c6b56ef16" path="/var/lib/kubelet/pods/6b4e2373-e7d6-451d-9213-257c6b56ef16/volumes"
Oct 30 13:29:34.262236 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:34.262200    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4091a5-59f7cb6b7f-7472v"]
Oct 30 13:29:35.478597 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:35.478566    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-167965-fdb89d64c-td5pc"
Oct 30 13:29:35.478987 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:35.478633    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-167965-6b479587d8-zktwq"
Oct 30 13:29:39.058993 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:39.058950    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-167965-74ddd66c88-rw9pw"]
Oct 30 13:29:39.059974 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:39.059945    5093 cpu_manager.go:414] "containerMap: removing container" podUID="6b4e2373-e7d6-451d-9213-257c6b56ef16" containerName="maas-api"
Oct 30 13:29:39.060063 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:39.059976    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b4e2373-e7d6-451d-9213-257c6b56ef16" containerName="maas-api"
Oct 30 13:29:39.060191 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:39.060178    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b4e2373-e7d6-451d-9213-257c6b56ef16" containerName="maas-api"
Oct 30 13:29:39.068760 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:39.068731    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-167965-74ddd66c88-rw9pw"
Oct 30 13:29:39.080067 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:39.080033    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-167965-74ddd66c88-rw9pw"]
Oct 30 13:29:39.089406 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:39.089379    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-167965-serving-cert\""
Oct 30 13:29:39.211398 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:39.211361    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17e5bb80-effa-4c06-b94f-b7e62b372ea8-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-167965-74ddd66c88-rw9pw\" (UID: \"17e5bb80-effa-4c06-b94f-b7e62b372ea8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-167965-74ddd66c88-rw9pw"
Oct 30 13:29:39.211398 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:39.211400    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54rtk\" (UniqueName: \"kubernetes.io/projected/17e5bb80-effa-4c06-b94f-b7e62b372ea8-kube-api-access-54rtk\") pod \"maas-api-e2e-shared-a-w4-167965-74ddd66c88-rw9pw\" (UID: \"17e5bb80-effa-4c06-b94f-b7e62b372ea8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-167965-74ddd66c88-rw9pw"
Oct 30 13:29:39.211644 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:39.211429    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17e5bb80-effa-4c06-b94f-b7e62b372ea8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-167965-74ddd66c88-rw9pw\" (UID: \"17e5bb80-effa-4c06-b94f-b7e62b372ea8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-167965-74ddd66c88-rw9pw"
Oct 30 13:29:39.211644 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:39.211566    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17e5bb80-effa-4c06-b94f-b7e62b372ea8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-167965-74ddd66c88-rw9pw\" (UID: \"17e5bb80-effa-4c06-b94f-b7e62b372ea8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-167965-74ddd66c88-rw9pw"
Oct 30 13:29:39.263058 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:39.262965    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4091a5-59f7cb6b7f-7472v" podUID="30752e68-5a72-423a-8afe-9793414d0c1f" containerName="payload-pre-processing" containerID="cri-o://2aa7e0bf100ba2594769ea6703b2d1fc80307b7b91155c4fdd80f4070059d632" gracePeriod=25
Oct 30 13:29:39.312964 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:39.312870    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17e5bb80-effa-4c06-b94f-b7e62b372ea8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-167965-74ddd66c88-rw9pw\" (UID: \"17e5bb80-effa-4c06-b94f-b7e62b372ea8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-167965-74ddd66c88-rw9pw"
Oct 30 13:29:39.313238 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:39.313219    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17e5bb80-effa-4c06-b94f-b7e62b372ea8-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-167965-74ddd66c88-rw9pw\" (UID: \"17e5bb80-effa-4c06-b94f-b7e62b372ea8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-167965-74ddd66c88-rw9pw"
Oct 30 13:29:39.313344 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:39.313251    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54rtk\" (UniqueName: \"kubernetes.io/projected/17e5bb80-effa-4c06-b94f-b7e62b372ea8-kube-api-access-54rtk\") pod \"maas-api-e2e-shared-a-w4-167965-74ddd66c88-rw9pw\" (UID: \"17e5bb80-effa-4c06-b94f-b7e62b372ea8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-167965-74ddd66c88-rw9pw"
Oct 30 13:29:39.313344 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:39.313278    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17e5bb80-effa-4c06-b94f-b7e62b372ea8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-167965-74ddd66c88-rw9pw\" (UID: \"17e5bb80-effa-4c06-b94f-b7e62b372ea8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-167965-74ddd66c88-rw9pw"
Oct 30 13:29:39.313887 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:39.313860    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17e5bb80-effa-4c06-b94f-b7e62b372ea8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-167965-74ddd66c88-rw9pw\" (UID: \"17e5bb80-effa-4c06-b94f-b7e62b372ea8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-167965-74ddd66c88-rw9pw"
Oct 30 13:29:39.315570 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:39.315551    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17e5bb80-effa-4c06-b94f-b7e62b372ea8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-167965-74ddd66c88-rw9pw\" (UID: \"17e5bb80-effa-4c06-b94f-b7e62b372ea8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-167965-74ddd66c88-rw9pw"
Oct 30 13:29:39.315649 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:39.315597    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17e5bb80-effa-4c06-b94f-b7e62b372ea8-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-167965-74ddd66c88-rw9pw\" (UID: \"17e5bb80-effa-4c06-b94f-b7e62b372ea8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-167965-74ddd66c88-rw9pw"
Oct 30 13:29:39.333871 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:39.333844    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54rtk\" (UniqueName: \"kubernetes.io/projected/17e5bb80-effa-4c06-b94f-b7e62b372ea8-kube-api-access-54rtk\") pod \"maas-api-e2e-shared-a-w4-167965-74ddd66c88-rw9pw\" (UID: \"17e5bb80-effa-4c06-b94f-b7e62b372ea8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-167965-74ddd66c88-rw9pw"
Oct 30 13:29:39.393481 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:39.393451    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-167965-74ddd66c88-rw9pw"
Oct 30 13:29:39.580876 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:39.571303    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cab6c5-866cf94959-h48bz"
Oct 30 13:29:39.580876 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:39.576341    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cab6c5-555d87bc58-6slkr"
Oct 30 13:29:39.651633 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:39.651588    5093 generic.go:356] "Generic (PLEG): container finished" podID="30752e68-5a72-423a-8afe-9793414d0c1f" containerID="2aa7e0bf100ba2594769ea6703b2d1fc80307b7b91155c4fdd80f4070059d632" exitCode=0
Oct 30 13:29:39.652001 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:39.651725    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4091a5-59f7cb6b7f-7472v" event={"ID":"30752e68-5a72-423a-8afe-9793414d0c1f","Type":"ContainerDied","Data":"2aa7e0bf100ba2594769ea6703b2d1fc80307b7b91155c4fdd80f4070059d632"}
Oct 30 13:29:39.789093 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:39.789060    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4091a5-59f7cb6b7f-7472v"
Oct 30 13:29:39.792707 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:39.792660    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-167965-74ddd66c88-rw9pw"]
Oct 30 13:29:39.793187 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:29:39.793102    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17e5bb80_effa_4c06_b94f_b7e62b372ea8.slice/crio-b3f50f9455f1904819d705550ca3d918e4051b368d52276bc1c9d4e9190aeb7e WatchSource:0}: Error finding container b3f50f9455f1904819d705550ca3d918e4051b368d52276bc1c9d4e9190aeb7e: Status 404 returned error can't find the container with id b3f50f9455f1904819d705550ca3d918e4051b368d52276bc1c9d4e9190aeb7e
Oct 30 13:29:39.795355 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:39.795336    5093 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 13:29:39.967226 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:39.967162    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30752e68-5a72-423a-8afe-9793414d0c1f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30752e68-5a72-423a-8afe-9793414d0c1f-plugins-config-volume\") pod \"30752e68-5a72-423a-8afe-9793414d0c1f\" (UID: \"30752e68-5a72-423a-8afe-9793414d0c1f\") "
Oct 30 13:29:39.967226 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:39.967211    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30752e68-5a72-423a-8afe-9793414d0c1f-kube-api-access-t8dfs\" (UniqueName: \"kubernetes.io/projected/30752e68-5a72-423a-8afe-9793414d0c1f-kube-api-access-t8dfs\") pod \"30752e68-5a72-423a-8afe-9793414d0c1f\" (UID: \"30752e68-5a72-423a-8afe-9793414d0c1f\") "
Oct 30 13:29:39.967595 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:39.967568    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30752e68-5a72-423a-8afe-9793414d0c1f-plugins-config-volume" pod "30752e68-5a72-423a-8afe-9793414d0c1f" (UID: "30752e68-5a72-423a-8afe-9793414d0c1f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:29:39.967843 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:39.967822    5093 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30752e68-5a72-423a-8afe-9793414d0c1f-plugins-config-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:29:39.978680 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:39.978637    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30752e68-5a72-423a-8afe-9793414d0c1f-kube-api-access-t8dfs" pod "30752e68-5a72-423a-8afe-9793414d0c1f" (UID: "30752e68-5a72-423a-8afe-9793414d0c1f"). InnerVolumeSpecName "kube-api-access-t8dfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:29:40.068932 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:40.068895    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8dfs\" (UniqueName: \"kubernetes.io/projected/30752e68-5a72-423a-8afe-9793414d0c1f-kube-api-access-t8dfs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:29:40.661011 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:40.660911    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-167965-74ddd66c88-rw9pw" event={"ID":"17e5bb80-effa-4c06-b94f-b7e62b372ea8","Type":"ContainerStarted","Data":"84c3cb33553972cdbbb778d577d575dc540fe57b9809880e00a72ec79b3455b9"}
Oct 30 13:29:40.661011 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:40.660987    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-167965-74ddd66c88-rw9pw"
Oct 30 13:29:40.661245 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:40.661016    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-167965-74ddd66c88-rw9pw" event={"ID":"17e5bb80-effa-4c06-b94f-b7e62b372ea8","Type":"ContainerStarted","Data":"b3f50f9455f1904819d705550ca3d918e4051b368d52276bc1c9d4e9190aeb7e"}
Oct 30 13:29:40.662480 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:40.662455    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4091a5-59f7cb6b7f-7472v"
Oct 30 13:29:40.662621 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:40.662471    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4091a5-59f7cb6b7f-7472v" event={"ID":"30752e68-5a72-423a-8afe-9793414d0c1f","Type":"ContainerDied","Data":"8bdc3ba0682405dc1f1185294932f07fe30437c07e36611123a3b6172b7a8388"}
Oct 30 13:29:40.662621 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:40.662572    5093 scope.go:122] "RemoveContainer" containerID="2aa7e0bf100ba2594769ea6703b2d1fc80307b7b91155c4fdd80f4070059d632"
Oct 30 13:29:40.758027 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:40.757956    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-167965-74ddd66c88-rw9pw" podStartSLOduration=1.39173349 podStartE2EDuration="1.757935767s" totalImagesPullingTime="366.202277ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:40.71133435 +0000 UTC m=+1599.858601591" watchObservedRunningTime="2026-10-30 13:29:40.757935767 +0000 UTC m=+1599.905203010"
Oct 30 13:29:40.765001 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:40.760779    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4091a5-59f7cb6b7f-7472v"]
Oct 30 13:29:40.766841 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:40.766807    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4091a5-59f7cb6b7f-7472v"]
Oct 30 13:29:41.002737 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:41.002701    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30752e68-5a72-423a-8afe-9793414d0c1f" path="/var/lib/kubelet/pods/30752e68-5a72-423a-8afe-9793414d0c1f/volumes"
Oct 30 13:29:44.743117 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:44.743082    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cab6c5-8559b4f669-ktcqz"]
Oct 30 13:29:44.743947 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:44.743923    5093 cpu_manager.go:414] "containerMap: removing container" podUID="30752e68-5a72-423a-8afe-9793414d0c1f" containerName="payload-pre-processing"
Oct 30 13:29:44.744065 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:44.743948    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30752e68-5a72-423a-8afe-9793414d0c1f" containerName="payload-pre-processing"
Oct 30 13:29:44.744166 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:44.744148    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="30752e68-5a72-423a-8afe-9793414d0c1f" containerName="payload-pre-processing"
Oct 30 13:29:44.749075 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:44.749053    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cab6c5-8559b4f669-ktcqz"
Oct 30 13:29:44.765712 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:44.765684    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cab6c5-8559b4f669-ktcqz"]
Oct 30 13:29:44.774284 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:44.774254    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-cab6c5-serving-cert\""
Oct 30 13:29:44.839826 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:44.839775    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slq69\" (UniqueName: \"kubernetes.io/projected/14d0074c-a3ab-4ec3-b8e8-95b4681260fd-kube-api-access-slq69\") pod \"maas-api-e2e-shared-a-w3-cab6c5-8559b4f669-ktcqz\" (UID: \"14d0074c-a3ab-4ec3-b8e8-95b4681260fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cab6c5-8559b4f669-ktcqz"
Oct 30 13:29:44.840038 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:44.839942    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14d0074c-a3ab-4ec3-b8e8-95b4681260fd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-cab6c5-8559b4f669-ktcqz\" (UID: \"14d0074c-a3ab-4ec3-b8e8-95b4681260fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cab6c5-8559b4f669-ktcqz"
Oct 30 13:29:44.840038 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:44.840000    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14d0074c-a3ab-4ec3-b8e8-95b4681260fd-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-cab6c5-8559b4f669-ktcqz\" (UID: \"14d0074c-a3ab-4ec3-b8e8-95b4681260fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cab6c5-8559b4f669-ktcqz"
Oct 30 13:29:44.840204 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:44.840121    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14d0074c-a3ab-4ec3-b8e8-95b4681260fd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-cab6c5-8559b4f669-ktcqz\" (UID: \"14d0074c-a3ab-4ec3-b8e8-95b4681260fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cab6c5-8559b4f669-ktcqz"
Oct 30 13:29:44.941225 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:44.941187    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slq69\" (UniqueName: \"kubernetes.io/projected/14d0074c-a3ab-4ec3-b8e8-95b4681260fd-kube-api-access-slq69\") pod \"maas-api-e2e-shared-a-w3-cab6c5-8559b4f669-ktcqz\" (UID: \"14d0074c-a3ab-4ec3-b8e8-95b4681260fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cab6c5-8559b4f669-ktcqz"
Oct 30 13:29:44.941413 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:44.941270    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14d0074c-a3ab-4ec3-b8e8-95b4681260fd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-cab6c5-8559b4f669-ktcqz\" (UID: \"14d0074c-a3ab-4ec3-b8e8-95b4681260fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cab6c5-8559b4f669-ktcqz"
Oct 30 13:29:44.941413 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:44.941292    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14d0074c-a3ab-4ec3-b8e8-95b4681260fd-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-cab6c5-8559b4f669-ktcqz\" (UID: \"14d0074c-a3ab-4ec3-b8e8-95b4681260fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cab6c5-8559b4f669-ktcqz"
Oct 30 13:29:44.941413 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:44.941361    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14d0074c-a3ab-4ec3-b8e8-95b4681260fd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-cab6c5-8559b4f669-ktcqz\" (UID: \"14d0074c-a3ab-4ec3-b8e8-95b4681260fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cab6c5-8559b4f669-ktcqz"
Oct 30 13:29:44.941603 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:29:44.941545    5093 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cab6c5-serving-cert: secret "maas-api-e2e-shared-a-w3-cab6c5-serving-cert" not found
Oct 30 13:29:44.941669 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:29:44.941613    5093 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14d0074c-a3ab-4ec3-b8e8-95b4681260fd-maas-api-tls podName:14d0074c-a3ab-4ec3-b8e8-95b4681260fd nodeName:}" failed. No retries permitted until 2026-10-30 13:29:45.441591491 +0000 UTC m=+1604.588858711 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/14d0074c-a3ab-4ec3-b8e8-95b4681260fd-maas-api-tls") pod "maas-api-e2e-shared-a-w3-cab6c5-8559b4f669-ktcqz" (UID: "14d0074c-a3ab-4ec3-b8e8-95b4681260fd") : secret "maas-api-e2e-shared-a-w3-cab6c5-serving-cert" not found
Oct 30 13:29:44.942053 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:44.942025    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14d0074c-a3ab-4ec3-b8e8-95b4681260fd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-cab6c5-8559b4f669-ktcqz\" (UID: \"14d0074c-a3ab-4ec3-b8e8-95b4681260fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cab6c5-8559b4f669-ktcqz"
Oct 30 13:29:44.943948 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:44.943921    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14d0074c-a3ab-4ec3-b8e8-95b4681260fd-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-cab6c5-8559b4f669-ktcqz\" (UID: \"14d0074c-a3ab-4ec3-b8e8-95b4681260fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cab6c5-8559b4f669-ktcqz"
Oct 30 13:29:44.962473 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:44.962437    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slq69\" (UniqueName: \"kubernetes.io/projected/14d0074c-a3ab-4ec3-b8e8-95b4681260fd-kube-api-access-slq69\") pod \"maas-api-e2e-shared-a-w3-cab6c5-8559b4f669-ktcqz\" (UID: \"14d0074c-a3ab-4ec3-b8e8-95b4681260fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cab6c5-8559b4f669-ktcqz"
Oct 30 13:29:45.448751 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:45.448712    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14d0074c-a3ab-4ec3-b8e8-95b4681260fd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-cab6c5-8559b4f669-ktcqz\" (UID: \"14d0074c-a3ab-4ec3-b8e8-95b4681260fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cab6c5-8559b4f669-ktcqz"
Oct 30 13:29:45.451361 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:45.451333    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14d0074c-a3ab-4ec3-b8e8-95b4681260fd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-cab6c5-8559b4f669-ktcqz\" (UID: \"14d0074c-a3ab-4ec3-b8e8-95b4681260fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cab6c5-8559b4f669-ktcqz"
Oct 30 13:29:45.664454 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:45.664407    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cab6c5-8559b4f669-ktcqz"
Oct 30 13:29:45.969879 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:29:45.969844    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14d0074c_a3ab_4ec3_b8e8_95b4681260fd.slice/crio-0873e79b6406e53887f57f81c3c162d2771eb8a8315af97a3ae33ce67f451756 WatchSource:0}: Error finding container 0873e79b6406e53887f57f81c3c162d2771eb8a8315af97a3ae33ce67f451756: Status 404 returned error can't find the container with id 0873e79b6406e53887f57f81c3c162d2771eb8a8315af97a3ae33ce67f451756
Oct 30 13:29:45.971987 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:45.971922    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cab6c5-8559b4f669-ktcqz"]
Oct 30 13:29:46.676795 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:46.676761    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-167965-74ddd66c88-rw9pw"
Oct 30 13:29:46.714635 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:46.714596    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cab6c5-8559b4f669-ktcqz" event={"ID":"14d0074c-a3ab-4ec3-b8e8-95b4681260fd","Type":"ContainerStarted","Data":"f9e314a6be63c6c95b6952d98b38ff6363c2e3e9c27dde69d356a911807d94ca"}
Oct 30 13:29:46.714843 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:46.714651    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cab6c5-8559b4f669-ktcqz" event={"ID":"14d0074c-a3ab-4ec3-b8e8-95b4681260fd","Type":"ContainerStarted","Data":"0873e79b6406e53887f57f81c3c162d2771eb8a8315af97a3ae33ce67f451756"}
Oct 30 13:29:46.714843 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:46.714694    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cab6c5-8559b4f669-ktcqz"
Oct 30 13:29:46.764708 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:46.764652    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cab6c5-8559b4f669-ktcqz" podStartSLOduration=2.438626579 podStartE2EDuration="2.764636165s" totalImagesPullingTime="326.009586ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:46.759760822 +0000 UTC m=+1605.907028095" watchObservedRunningTime="2026-10-30 13:29:46.764636165 +0000 UTC m=+1605.911903406"
Oct 30 13:29:51.170340 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:29:51.170293    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:29:52.731412 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:29:52.731379    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cab6c5-8559b4f669-ktcqz"
Oct 30 13:30:10.958603 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:10.958567    5093 generic.go:356] "Generic (PLEG): container finished" podID="1a9a0d53-3fb0-414a-a543-f5dd4ed39eef" containerID="e6d77701831809fd79f318075f38754db29a881178681c1154bc41ee91ddc86f" exitCode=0
Oct 30 13:30:10.959064 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:10.958640    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x" event={"ID":"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef","Type":"ContainerDied","Data":"e6d77701831809fd79f318075f38754db29a881178681c1154bc41ee91ddc86f"}
Oct 30 13:30:10.959234 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:10.959217    5093 scope.go:122] "RemoveContainer" containerID="e6d77701831809fd79f318075f38754db29a881178681c1154bc41ee91ddc86f"
Oct 30 13:30:10.960851 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:10.960821    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:30:10.960953 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:10.960895    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:30:10.960953 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:10.960930    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:30:11.971544 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:11.971499    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x" event={"ID":"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef","Type":"ContainerStarted","Data":"490b4fae81b4f157e902505c0a49e4e7eadb9bd26ef722a0a577ca88ccbc80a5"}
Oct 30 13:30:12.807755 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:12.805465    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x"
Oct 30 13:30:12.807755 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:12.805575    5093 patch_prober.go:29] interesting pod/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 30 13:30:12.807755 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:12.805623    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x" podUID="1a9a0d53-3fb0-414a-a543-f5dd4ed39eef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 30 13:30:13.804360 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:13.804322    5093 patch_prober.go:29] interesting pod/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 30 13:30:13.804871 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:13.804388    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x" podUID="1a9a0d53-3fb0-414a-a543-f5dd4ed39eef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 30 13:30:14.803560 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:14.803521    5093 patch_prober.go:29] interesting pod/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 30 13:30:14.803746 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:14.803591    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x" podUID="1a9a0d53-3fb0-414a-a543-f5dd4ed39eef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 30 13:30:14.816479 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:14.816445    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f12b80-57dc694575-72wmr"]
Oct 30 13:30:14.822455 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:14.822431    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f12b80-57dc694575-72wmr"
Oct 30 13:30:14.850153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:14.849294    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f12b80-57dc694575-72wmr"]
Oct 30 13:30:14.859425 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:14.856088    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-f12b80\""
Oct 30 13:30:14.859425 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:14.856892    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-f12b80-dockercfg-4nq24\""
Oct 30 13:30:14.916062 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:14.916027    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a707991-287b-4944-a992-72e750b13a89-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-f12b80-57dc694575-72wmr\" (UID: \"3a707991-287b-4944-a992-72e750b13a89\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f12b80-57dc694575-72wmr"
Oct 30 13:30:14.916288 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:14.916245    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv89t\" (UniqueName: \"kubernetes.io/projected/3a707991-287b-4944-a992-72e750b13a89-kube-api-access-lv89t\") pod \"payload-pre-processing-e2e-shared-b-w4-f12b80-57dc694575-72wmr\" (UID: \"3a707991-287b-4944-a992-72e750b13a89\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f12b80-57dc694575-72wmr"
Oct 30 13:30:14.972281 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:14.969228    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-f12b80-5dbbcf49b-tqx8w"]
Oct 30 13:30:14.975393 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:14.974871    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f12b80-5dbbcf49b-tqx8w"
Oct 30 13:30:15.005263 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:15.005171    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-f12b80-5dbbcf49b-tqx8w"]
Oct 30 13:30:15.020414 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:15.020349    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b8aed42-a18f-4cb3-aa5d-7a3cf0345391-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-f12b80-5dbbcf49b-tqx8w\" (UID: \"4b8aed42-a18f-4cb3-aa5d-7a3cf0345391\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f12b80-5dbbcf49b-tqx8w"
Oct 30 13:30:15.020671 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:15.020474    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a707991-287b-4944-a992-72e750b13a89-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-f12b80-57dc694575-72wmr\" (UID: \"3a707991-287b-4944-a992-72e750b13a89\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f12b80-57dc694575-72wmr"
Oct 30 13:30:15.020671 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:15.020583    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv89t\" (UniqueName: \"kubernetes.io/projected/3a707991-287b-4944-a992-72e750b13a89-kube-api-access-lv89t\") pod \"payload-pre-processing-e2e-shared-b-w4-f12b80-57dc694575-72wmr\" (UID: \"3a707991-287b-4944-a992-72e750b13a89\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f12b80-57dc694575-72wmr"
Oct 30 13:30:15.020821 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:15.020761    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kznq6\" (UniqueName: \"kubernetes.io/projected/4b8aed42-a18f-4cb3-aa5d-7a3cf0345391-kube-api-access-kznq6\") pod \"payload-processing-e2e-shared-b-w4-f12b80-5dbbcf49b-tqx8w\" (UID: \"4b8aed42-a18f-4cb3-aa5d-7a3cf0345391\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f12b80-5dbbcf49b-tqx8w"
Oct 30 13:30:15.022096 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:15.022070    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a707991-287b-4944-a992-72e750b13a89-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-f12b80-57dc694575-72wmr\" (UID: \"3a707991-287b-4944-a992-72e750b13a89\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f12b80-57dc694575-72wmr"
Oct 30 13:30:15.064202 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:15.063750    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv89t\" (UniqueName: \"kubernetes.io/projected/3a707991-287b-4944-a992-72e750b13a89-kube-api-access-lv89t\") pod \"payload-pre-processing-e2e-shared-b-w4-f12b80-57dc694575-72wmr\" (UID: \"3a707991-287b-4944-a992-72e750b13a89\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f12b80-57dc694575-72wmr"
Oct 30 13:30:15.135152 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:15.125777    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kznq6\" (UniqueName: \"kubernetes.io/projected/4b8aed42-a18f-4cb3-aa5d-7a3cf0345391-kube-api-access-kznq6\") pod \"payload-processing-e2e-shared-b-w4-f12b80-5dbbcf49b-tqx8w\" (UID: \"4b8aed42-a18f-4cb3-aa5d-7a3cf0345391\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f12b80-5dbbcf49b-tqx8w"
Oct 30 13:30:15.135152 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:15.125964    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b8aed42-a18f-4cb3-aa5d-7a3cf0345391-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-f12b80-5dbbcf49b-tqx8w\" (UID: \"4b8aed42-a18f-4cb3-aa5d-7a3cf0345391\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f12b80-5dbbcf49b-tqx8w"
Oct 30 13:30:15.135152 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:15.127230    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b8aed42-a18f-4cb3-aa5d-7a3cf0345391-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-f12b80-5dbbcf49b-tqx8w\" (UID: \"4b8aed42-a18f-4cb3-aa5d-7a3cf0345391\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f12b80-5dbbcf49b-tqx8w"
Oct 30 13:30:15.170642 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:15.170606    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kznq6\" (UniqueName: \"kubernetes.io/projected/4b8aed42-a18f-4cb3-aa5d-7a3cf0345391-kube-api-access-kznq6\") pod \"payload-processing-e2e-shared-b-w4-f12b80-5dbbcf49b-tqx8w\" (UID: \"4b8aed42-a18f-4cb3-aa5d-7a3cf0345391\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f12b80-5dbbcf49b-tqx8w"
Oct 30 13:30:15.195789 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:15.195752    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f12b80-57dc694575-72wmr"
Oct 30 13:30:15.302530 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:15.301160    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f12b80-5dbbcf49b-tqx8w"
Oct 30 13:30:15.679879 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:30:15.679844    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a707991_287b_4944_a992_72e750b13a89.slice/crio-cd458aee00e90cdef204c5c5d458bc04b5df11f6764b5b865954442e42675049 WatchSource:0}: Error finding container cd458aee00e90cdef204c5c5d458bc04b5df11f6764b5b865954442e42675049: Status 404 returned error can't find the container with id cd458aee00e90cdef204c5c5d458bc04b5df11f6764b5b865954442e42675049
Oct 30 13:30:15.681733 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:15.681557    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f12b80-57dc694575-72wmr"]
Oct 30 13:30:15.719694 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:30:15.719663    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b8aed42_a18f_4cb3_aa5d_7a3cf0345391.slice/crio-98b7f41c2c9142dd3dae4f97f0e8b175b301021889f5f7e742cb8c0c58f4e079 WatchSource:0}: Error finding container 98b7f41c2c9142dd3dae4f97f0e8b175b301021889f5f7e742cb8c0c58f4e079: Status 404 returned error can't find the container with id 98b7f41c2c9142dd3dae4f97f0e8b175b301021889f5f7e742cb8c0c58f4e079
Oct 30 13:30:15.720324 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:15.720299    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-f12b80-5dbbcf49b-tqx8w"]
Oct 30 13:30:15.805197 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:15.803663    5093 patch_prober.go:29] interesting pod/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 30 13:30:15.805197 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:15.803714    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x" podUID="1a9a0d53-3fb0-414a-a543-f5dd4ed39eef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 30 13:30:16.073296 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:16.073177    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f12b80-5dbbcf49b-tqx8w" event={"ID":"4b8aed42-a18f-4cb3-aa5d-7a3cf0345391","Type":"ContainerStarted","Data":"0e3f6d068a4edc1a11e2bf27491b93ab500201a6664122b68680d843db098aee"}
Oct 30 13:30:16.073296 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:16.073262    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f12b80-5dbbcf49b-tqx8w" event={"ID":"4b8aed42-a18f-4cb3-aa5d-7a3cf0345391","Type":"ContainerStarted","Data":"98b7f41c2c9142dd3dae4f97f0e8b175b301021889f5f7e742cb8c0c58f4e079"}
Oct 30 13:30:16.074582 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:16.074554    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f12b80-5dbbcf49b-tqx8w"
Oct 30 13:30:16.077076 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:16.077037    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f12b80-57dc694575-72wmr" event={"ID":"3a707991-287b-4944-a992-72e750b13a89","Type":"ContainerStarted","Data":"1b791ef6b49e5ee40648df4827c9844c9675b705f7548a30a51315dab3ddf6b7"}
Oct 30 13:30:16.077232 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:16.077098    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f12b80-57dc694575-72wmr" event={"ID":"3a707991-287b-4944-a992-72e750b13a89","Type":"ContainerStarted","Data":"cd458aee00e90cdef204c5c5d458bc04b5df11f6764b5b865954442e42675049"}
Oct 30 13:30:16.077403 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:16.077382    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f12b80-57dc694575-72wmr"
Oct 30 13:30:16.124322 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:16.124271    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f12b80-5dbbcf49b-tqx8w" podStartSLOduration=2.124257317 podStartE2EDuration="2.124257317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:16.11981306 +0000 UTC m=+1635.267080298" watchObservedRunningTime="2026-10-30 13:30:16.124257317 +0000 UTC m=+1635.271524557"
Oct 30 13:30:16.803556 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:16.803521    5093 patch_prober.go:29] interesting pod/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 30 13:30:16.803743 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:16.803578    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x" podUID="1a9a0d53-3fb0-414a-a543-f5dd4ed39eef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 30 13:30:17.808063 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:17.808029    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x"
Oct 30 13:30:17.808651 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:17.808430    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x"
Oct 30 13:30:17.809114 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:17.809096    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x"
Oct 30 13:30:17.903767 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:17.903697    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f12b80-57dc694575-72wmr" podStartSLOduration=3.903681035 podStartE2EDuration="3.903681035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:16.180362678 +0000 UTC m=+1635.327629913" watchObservedRunningTime="2026-10-30 13:30:17.903681035 +0000 UTC m=+1637.050948283"
Oct 30 13:30:20.928153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:20.928084    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-f7ee8a-586db84f58-bfjng"]
Oct 30 13:30:21.232203 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:21.226354    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f7ee8a-586db84f58-bfjng"
Oct 30 13:30:21.299755 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:21.299399    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-f7ee8a\""
Oct 30 13:30:21.299755 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:21.299500    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-f7ee8a-dockercfg-8nr4j\""
Oct 30 13:30:21.300762 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:21.300721    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f88008e4-5f1c-4814-94aa-619e90c5fd0c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-f7ee8a-586db84f58-bfjng\" (UID: \"f88008e4-5f1c-4814-94aa-619e90c5fd0c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f7ee8a-586db84f58-bfjng"
Oct 30 13:30:21.300923 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:21.300894    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd682\" (UniqueName: \"kubernetes.io/projected/f88008e4-5f1c-4814-94aa-619e90c5fd0c-kube-api-access-vd682\") pod \"payload-processing-e2e-shared-a-w1-f7ee8a-586db84f58-bfjng\" (UID: \"f88008e4-5f1c-4814-94aa-619e90c5fd0c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f7ee8a-586db84f58-bfjng"
Oct 30 13:30:21.304760 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:21.304728    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-f7ee8a-586db84f58-bfjng"]
Oct 30 13:30:21.360491 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:30:21.359845    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:30:21.403743 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:21.403701    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f88008e4-5f1c-4814-94aa-619e90c5fd0c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-f7ee8a-586db84f58-bfjng\" (UID: \"f88008e4-5f1c-4814-94aa-619e90c5fd0c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f7ee8a-586db84f58-bfjng"
Oct 30 13:30:21.404090 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:21.404073    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd682\" (UniqueName: \"kubernetes.io/projected/f88008e4-5f1c-4814-94aa-619e90c5fd0c-kube-api-access-vd682\") pod \"payload-processing-e2e-shared-a-w1-f7ee8a-586db84f58-bfjng\" (UID: \"f88008e4-5f1c-4814-94aa-619e90c5fd0c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f7ee8a-586db84f58-bfjng"
Oct 30 13:30:21.406520 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:21.404666    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f88008e4-5f1c-4814-94aa-619e90c5fd0c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-f7ee8a-586db84f58-bfjng\" (UID: \"f88008e4-5f1c-4814-94aa-619e90c5fd0c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f7ee8a-586db84f58-bfjng"
Oct 30 13:30:21.504757 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:21.504671    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd682\" (UniqueName: \"kubernetes.io/projected/f88008e4-5f1c-4814-94aa-619e90c5fd0c-kube-api-access-vd682\") pod \"payload-processing-e2e-shared-a-w1-f7ee8a-586db84f58-bfjng\" (UID: \"f88008e4-5f1c-4814-94aa-619e90c5fd0c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f7ee8a-586db84f58-bfjng"
Oct 30 13:30:21.692621 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:21.692585    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f7ee8a-586db84f58-bfjng"
Oct 30 13:30:22.256644 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:22.256604    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-f7ee8a-586db84f58-bfjng"]
Oct 30 13:30:22.257090 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:30:22.256973    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf88008e4_5f1c_4814_94aa_619e90c5fd0c.slice/crio-3c4b663410b896fcabf425405b0280c301ef7581ebec29d1d34274d6918814fb WatchSource:0}: Error finding container 3c4b663410b896fcabf425405b0280c301ef7581ebec29d1d34274d6918814fb: Status 404 returned error can't find the container with id 3c4b663410b896fcabf425405b0280c301ef7581ebec29d1d34274d6918814fb
Oct 30 13:30:23.262322 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:23.262281    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f7ee8a-586db84f58-bfjng" event={"ID":"f88008e4-5f1c-4814-94aa-619e90c5fd0c","Type":"ContainerStarted","Data":"0e328bf699d341231dbbe138f3bbb701a6468a7c49f73a98031121d4041a587c"}
Oct 30 13:30:23.262893 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:23.262866    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f7ee8a-586db84f58-bfjng" event={"ID":"f88008e4-5f1c-4814-94aa-619e90c5fd0c","Type":"ContainerStarted","Data":"3c4b663410b896fcabf425405b0280c301ef7581ebec29d1d34274d6918814fb"}
Oct 30 13:30:23.264166 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:23.264142    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f7ee8a-586db84f58-bfjng"
Oct 30 13:30:23.315314 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:23.315263    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f7ee8a-586db84f58-bfjng" podStartSLOduration=3.315249303 podStartE2EDuration="3.315249303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:23.312553905 +0000 UTC m=+1642.459821146" watchObservedRunningTime="2026-10-30 13:30:23.315249303 +0000 UTC m=+1642.462516544"
Oct 30 13:30:27.090340 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:27.090307    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f12b80-57dc694575-72wmr"
Oct 30 13:30:28.096460 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:28.096428    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f12b80-5dbbcf49b-tqx8w"
Oct 30 13:30:31.328102 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:31.328069    5093 generic.go:356] "Generic (PLEG): container finished" podID="cc75f0e6-64c6-4fde-9476-3859350da986" containerID="b2c240320d8110c9d359ce33e13d8f8d1fde4db9e02e3c491dab4ca21fff450b" exitCode=0
Oct 30 13:30:31.328699 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:31.328162    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp" event={"ID":"cc75f0e6-64c6-4fde-9476-3859350da986","Type":"ContainerDied","Data":"b2c240320d8110c9d359ce33e13d8f8d1fde4db9e02e3c491dab4ca21fff450b"}
Oct 30 13:30:31.328778 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:31.328721    5093 scope.go:122] "RemoveContainer" containerID="b2c240320d8110c9d359ce33e13d8f8d1fde4db9e02e3c491dab4ca21fff450b"
Oct 30 13:30:31.330437 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:31.330402    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:30:31.330539 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:31.330477    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:30:31.330539 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:31.330508    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:30:32.358537 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:32.358499    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp" event={"ID":"cc75f0e6-64c6-4fde-9476-3859350da986","Type":"ContainerStarted","Data":"9826755f30199716a4db91c54e8186873daa229ff6f3eb83d664150033f7454f"}
Oct 30 13:30:32.511632 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:32.511590    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp"
Oct 30 13:30:32.511890 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:32.511855    5093 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 30 13:30:32.511993 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:32.511916    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp" podUID="cc75f0e6-64c6-4fde-9476-3859350da986" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 30 13:30:33.511835 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:33.511797    5093 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 30 13:30:33.512429 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:33.512382    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp" podUID="cc75f0e6-64c6-4fde-9476-3859350da986" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 30 13:30:34.511718 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:34.511625    5093 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 30 13:30:34.511718 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:34.511704    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp" podUID="cc75f0e6-64c6-4fde-9476-3859350da986" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 30 13:30:34.903877 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:34.903793    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f12b80-6c694985fd-g8lxp"]
Oct 30 13:30:34.909657 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:34.909631    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f12b80-6c694985fd-g8lxp"
Oct 30 13:30:34.944823 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:34.944791    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-f12b80-serving-cert\""
Oct 30 13:30:34.952630 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:34.952579    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f12b80-6c694985fd-g8lxp"]
Oct 30 13:30:34.987966 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:34.987933    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzd7r\" (UniqueName: \"kubernetes.io/projected/0d63ba90-2e39-44a9-aa12-ec1adf2dc23f-kube-api-access-mzd7r\") pod \"maas-api-e2e-shared-b-w4-f12b80-6c694985fd-g8lxp\" (UID: \"0d63ba90-2e39-44a9-aa12-ec1adf2dc23f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f12b80-6c694985fd-g8lxp"
Oct 30 13:30:34.988177 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:34.987991    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d63ba90-2e39-44a9-aa12-ec1adf2dc23f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-f12b80-6c694985fd-g8lxp\" (UID: \"0d63ba90-2e39-44a9-aa12-ec1adf2dc23f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f12b80-6c694985fd-g8lxp"
Oct 30 13:30:34.988177 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:34.988050    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d63ba90-2e39-44a9-aa12-ec1adf2dc23f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-f12b80-6c694985fd-g8lxp\" (UID: \"0d63ba90-2e39-44a9-aa12-ec1adf2dc23f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f12b80-6c694985fd-g8lxp"
Oct 30 13:30:34.988313 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:34.988274    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d63ba90-2e39-44a9-aa12-ec1adf2dc23f-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-f12b80-6c694985fd-g8lxp\" (UID: \"0d63ba90-2e39-44a9-aa12-ec1adf2dc23f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f12b80-6c694985fd-g8lxp"
Oct 30 13:30:35.091161 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:35.091094    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzd7r\" (UniqueName: \"kubernetes.io/projected/0d63ba90-2e39-44a9-aa12-ec1adf2dc23f-kube-api-access-mzd7r\") pod \"maas-api-e2e-shared-b-w4-f12b80-6c694985fd-g8lxp\" (UID: \"0d63ba90-2e39-44a9-aa12-ec1adf2dc23f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f12b80-6c694985fd-g8lxp"
Oct 30 13:30:35.091161 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:35.091167    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d63ba90-2e39-44a9-aa12-ec1adf2dc23f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-f12b80-6c694985fd-g8lxp\" (UID: \"0d63ba90-2e39-44a9-aa12-ec1adf2dc23f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f12b80-6c694985fd-g8lxp"
Oct 30 13:30:35.091417 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:35.091216    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d63ba90-2e39-44a9-aa12-ec1adf2dc23f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-f12b80-6c694985fd-g8lxp\" (UID: \"0d63ba90-2e39-44a9-aa12-ec1adf2dc23f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f12b80-6c694985fd-g8lxp"
Oct 30 13:30:35.091417 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:35.091338    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d63ba90-2e39-44a9-aa12-ec1adf2dc23f-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-f12b80-6c694985fd-g8lxp\" (UID: \"0d63ba90-2e39-44a9-aa12-ec1adf2dc23f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f12b80-6c694985fd-g8lxp"
Oct 30 13:30:35.092484 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:30:35.092246    5093 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f12b80-serving-cert: secret "maas-api-e2e-shared-b-w4-f12b80-serving-cert" not found
Oct 30 13:30:35.092484 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:30:35.092316    5093 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d63ba90-2e39-44a9-aa12-ec1adf2dc23f-maas-api-tls podName:0d63ba90-2e39-44a9-aa12-ec1adf2dc23f nodeName:}" failed. No retries permitted until 2026-10-30 13:30:35.592294555 +0000 UTC m=+1654.739561777 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0d63ba90-2e39-44a9-aa12-ec1adf2dc23f-maas-api-tls") pod "maas-api-e2e-shared-b-w4-f12b80-6c694985fd-g8lxp" (UID: "0d63ba90-2e39-44a9-aa12-ec1adf2dc23f") : secret "maas-api-e2e-shared-b-w4-f12b80-serving-cert" not found
Oct 30 13:30:35.092958 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:35.092935    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d63ba90-2e39-44a9-aa12-ec1adf2dc23f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-f12b80-6c694985fd-g8lxp\" (UID: \"0d63ba90-2e39-44a9-aa12-ec1adf2dc23f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f12b80-6c694985fd-g8lxp"
Oct 30 13:30:35.095323 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:35.095297    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d63ba90-2e39-44a9-aa12-ec1adf2dc23f-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-f12b80-6c694985fd-g8lxp\" (UID: \"0d63ba90-2e39-44a9-aa12-ec1adf2dc23f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f12b80-6c694985fd-g8lxp"
Oct 30 13:30:35.118672 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:35.118646    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzd7r\" (UniqueName: \"kubernetes.io/projected/0d63ba90-2e39-44a9-aa12-ec1adf2dc23f-kube-api-access-mzd7r\") pod \"maas-api-e2e-shared-b-w4-f12b80-6c694985fd-g8lxp\" (UID: \"0d63ba90-2e39-44a9-aa12-ec1adf2dc23f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f12b80-6c694985fd-g8lxp"
Oct 30 13:30:35.283358 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:35.283329    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f7ee8a-586db84f58-bfjng"
Oct 30 13:30:35.511666 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:35.511630    5093 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 30 13:30:35.511921 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:35.511894    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp" podUID="cc75f0e6-64c6-4fde-9476-3859350da986" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 30 13:30:35.598493 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:35.598400    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d63ba90-2e39-44a9-aa12-ec1adf2dc23f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-f12b80-6c694985fd-g8lxp\" (UID: \"0d63ba90-2e39-44a9-aa12-ec1adf2dc23f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f12b80-6c694985fd-g8lxp"
Oct 30 13:30:35.603178 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:35.601159    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d63ba90-2e39-44a9-aa12-ec1adf2dc23f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-f12b80-6c694985fd-g8lxp\" (UID: \"0d63ba90-2e39-44a9-aa12-ec1adf2dc23f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f12b80-6c694985fd-g8lxp"
Oct 30 13:30:35.824506 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:35.824471    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f12b80-6c694985fd-g8lxp"
Oct 30 13:30:36.270525 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:36.270473    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f12b80-6c694985fd-g8lxp"]
Oct 30 13:30:36.401742 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:36.401705    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f12b80-6c694985fd-g8lxp" event={"ID":"0d63ba90-2e39-44a9-aa12-ec1adf2dc23f","Type":"ContainerStarted","Data":"bf89ec92b2b64e4469eca6f944ac1ff1f7de40540d19ced6e2c50c5f0d8f3b6d"}
Oct 30 13:30:36.511737 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:36.511653    5093 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 30 13:30:36.511737 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:36.511717    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp" podUID="cc75f0e6-64c6-4fde-9476-3859350da986" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 30 13:30:37.413722 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:37.413670    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f12b80-6c694985fd-g8lxp" event={"ID":"0d63ba90-2e39-44a9-aa12-ec1adf2dc23f","Type":"ContainerStarted","Data":"8d753f050bd7996a9aec9622c4b5309ced94f75bcb91015edac29b2baf598214"}
Oct 30 13:30:37.414310 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:37.413757    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f12b80-6c694985fd-g8lxp"
Oct 30 13:30:37.461613 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:37.461557    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f12b80-6c694985fd-g8lxp" podStartSLOduration=3.1535001559999998 podStartE2EDuration="3.461542899s" totalImagesPullingTime="308.042743ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:37.459125155 +0000 UTC m=+1656.606392409" watchObservedRunningTime="2026-10-30 13:30:37.461542899 +0000 UTC m=+1656.608810139"
Oct 30 13:30:37.587431 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:37.587399    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp"
Oct 30 13:30:37.588410 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:37.588381    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp"
Oct 30 13:30:37.589804 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:37.589778    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp"
Oct 30 13:30:38.078555 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:38.078513    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-c684d2/auth-test-model-c684d2-kserve-5cb979995c-6xppj"]
Oct 30 13:30:38.084239 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:38.084211    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-c684d2/auth-test-model-c684d2-kserve-5cb979995c-6xppj"
Oct 30 13:30:38.100899 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:38.100865    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-c684d2\"/\"openshift-service-ca.crt\""
Oct 30 13:30:38.100899 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:38.100863    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-c684d2\"/\"default-dockercfg-nkjpp\""
Oct 30 13:30:38.100899 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:38.100863    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-c684d2\"/\"auth-test-model-c684d2-kserve-self-signed-certs\""
Oct 30 13:30:38.102021 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:38.101998    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-c684d2/auth-test-model-c684d2-kserve-5cb979995c-6xppj"]
Oct 30 13:30:38.109939 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:38.109910    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-c684d2\"/\"kube-root-ca.crt\""
Oct 30 13:30:38.135591 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:38.135553    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08f03e92-3e35-4a93-a240-f38ed6572622-tmp-dir\") pod \"auth-test-model-c684d2-kserve-5cb979995c-6xppj\" (UID: \"08f03e92-3e35-4a93-a240-f38ed6572622\") " pod="ai-tenant-e2e-shared-a-w2-c684d2/auth-test-model-c684d2-kserve-5cb979995c-6xppj"
Oct 30 13:30:38.135591 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:38.135594    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08f03e92-3e35-4a93-a240-f38ed6572622-tls-certs\") pod \"auth-test-model-c684d2-kserve-5cb979995c-6xppj\" (UID: \"08f03e92-3e35-4a93-a240-f38ed6572622\") " pod="ai-tenant-e2e-shared-a-w2-c684d2/auth-test-model-c684d2-kserve-5cb979995c-6xppj"
Oct 30 13:30:38.135883 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:38.135727    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d66hg\" (UniqueName: \"kubernetes.io/projected/08f03e92-3e35-4a93-a240-f38ed6572622-kube-api-access-d66hg\") pod \"auth-test-model-c684d2-kserve-5cb979995c-6xppj\" (UID: \"08f03e92-3e35-4a93-a240-f38ed6572622\") " pod="ai-tenant-e2e-shared-a-w2-c684d2/auth-test-model-c684d2-kserve-5cb979995c-6xppj"
Oct 30 13:30:38.135883 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:38.135796    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08f03e92-3e35-4a93-a240-f38ed6572622-model-cache\") pod \"auth-test-model-c684d2-kserve-5cb979995c-6xppj\" (UID: \"08f03e92-3e35-4a93-a240-f38ed6572622\") " pod="ai-tenant-e2e-shared-a-w2-c684d2/auth-test-model-c684d2-kserve-5cb979995c-6xppj"
Oct 30 13:30:38.135883 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:38.135845    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08f03e92-3e35-4a93-a240-f38ed6572622-dshm\") pod \"auth-test-model-c684d2-kserve-5cb979995c-6xppj\" (UID: \"08f03e92-3e35-4a93-a240-f38ed6572622\") " pod="ai-tenant-e2e-shared-a-w2-c684d2/auth-test-model-c684d2-kserve-5cb979995c-6xppj"
Oct 30 13:30:38.136021 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:38.135890    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08f03e92-3e35-4a93-a240-f38ed6572622-home\") pod \"auth-test-model-c684d2-kserve-5cb979995c-6xppj\" (UID: \"08f03e92-3e35-4a93-a240-f38ed6572622\") " pod="ai-tenant-e2e-shared-a-w2-c684d2/auth-test-model-c684d2-kserve-5cb979995c-6xppj"
Oct 30 13:30:38.236791 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:38.236754    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08f03e92-3e35-4a93-a240-f38ed6572622-tmp-dir\") pod \"auth-test-model-c684d2-kserve-5cb979995c-6xppj\" (UID: \"08f03e92-3e35-4a93-a240-f38ed6572622\") " pod="ai-tenant-e2e-shared-a-w2-c684d2/auth-test-model-c684d2-kserve-5cb979995c-6xppj"
Oct 30 13:30:38.236791 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:38.236787    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08f03e92-3e35-4a93-a240-f38ed6572622-tls-certs\") pod \"auth-test-model-c684d2-kserve-5cb979995c-6xppj\" (UID: \"08f03e92-3e35-4a93-a240-f38ed6572622\") " pod="ai-tenant-e2e-shared-a-w2-c684d2/auth-test-model-c684d2-kserve-5cb979995c-6xppj"
Oct 30 13:30:38.237038 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:38.236860    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d66hg\" (UniqueName: \"kubernetes.io/projected/08f03e92-3e35-4a93-a240-f38ed6572622-kube-api-access-d66hg\") pod \"auth-test-model-c684d2-kserve-5cb979995c-6xppj\" (UID: \"08f03e92-3e35-4a93-a240-f38ed6572622\") " pod="ai-tenant-e2e-shared-a-w2-c684d2/auth-test-model-c684d2-kserve-5cb979995c-6xppj"
Oct 30 13:30:38.237038 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:38.236925    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08f03e92-3e35-4a93-a240-f38ed6572622-model-cache\") pod \"auth-test-model-c684d2-kserve-5cb979995c-6xppj\" (UID: \"08f03e92-3e35-4a93-a240-f38ed6572622\") " pod="ai-tenant-e2e-shared-a-w2-c684d2/auth-test-model-c684d2-kserve-5cb979995c-6xppj"
Oct 30 13:30:38.237038 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:38.236968    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08f03e92-3e35-4a93-a240-f38ed6572622-dshm\") pod \"auth-test-model-c684d2-kserve-5cb979995c-6xppj\" (UID: \"08f03e92-3e35-4a93-a240-f38ed6572622\") " pod="ai-tenant-e2e-shared-a-w2-c684d2/auth-test-model-c684d2-kserve-5cb979995c-6xppj"
Oct 30 13:30:38.237038 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:38.236992    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08f03e92-3e35-4a93-a240-f38ed6572622-home\") pod \"auth-test-model-c684d2-kserve-5cb979995c-6xppj\" (UID: \"08f03e92-3e35-4a93-a240-f38ed6572622\") " pod="ai-tenant-e2e-shared-a-w2-c684d2/auth-test-model-c684d2-kserve-5cb979995c-6xppj"
Oct 30 13:30:38.237273 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:38.237247    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08f03e92-3e35-4a93-a240-f38ed6572622-tmp-dir\") pod \"auth-test-model-c684d2-kserve-5cb979995c-6xppj\" (UID: \"08f03e92-3e35-4a93-a240-f38ed6572622\") " pod="ai-tenant-e2e-shared-a-w2-c684d2/auth-test-model-c684d2-kserve-5cb979995c-6xppj"
Oct 30 13:30:38.237399 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:38.237373    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08f03e92-3e35-4a93-a240-f38ed6572622-model-cache\") pod \"auth-test-model-c684d2-kserve-5cb979995c-6xppj\" (UID: \"08f03e92-3e35-4a93-a240-f38ed6572622\") " pod="ai-tenant-e2e-shared-a-w2-c684d2/auth-test-model-c684d2-kserve-5cb979995c-6xppj"
Oct 30 13:30:38.237484 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:38.237469    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08f03e92-3e35-4a93-a240-f38ed6572622-home\") pod \"auth-test-model-c684d2-kserve-5cb979995c-6xppj\" (UID: \"08f03e92-3e35-4a93-a240-f38ed6572622\") " pod="ai-tenant-e2e-shared-a-w2-c684d2/auth-test-model-c684d2-kserve-5cb979995c-6xppj"
Oct 30 13:30:38.239436 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:38.239406    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08f03e92-3e35-4a93-a240-f38ed6572622-dshm\") pod \"auth-test-model-c684d2-kserve-5cb979995c-6xppj\" (UID: \"08f03e92-3e35-4a93-a240-f38ed6572622\") " pod="ai-tenant-e2e-shared-a-w2-c684d2/auth-test-model-c684d2-kserve-5cb979995c-6xppj"
Oct 30 13:30:38.240007 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:38.239986    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08f03e92-3e35-4a93-a240-f38ed6572622-tls-certs\") pod \"auth-test-model-c684d2-kserve-5cb979995c-6xppj\" (UID: \"08f03e92-3e35-4a93-a240-f38ed6572622\") " pod="ai-tenant-e2e-shared-a-w2-c684d2/auth-test-model-c684d2-kserve-5cb979995c-6xppj"
Oct 30 13:30:38.257890 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:38.257863    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d66hg\" (UniqueName: \"kubernetes.io/projected/08f03e92-3e35-4a93-a240-f38ed6572622-kube-api-access-d66hg\") pod \"auth-test-model-c684d2-kserve-5cb979995c-6xppj\" (UID: \"08f03e92-3e35-4a93-a240-f38ed6572622\") " pod="ai-tenant-e2e-shared-a-w2-c684d2/auth-test-model-c684d2-kserve-5cb979995c-6xppj"
Oct 30 13:30:38.399343 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:38.399258    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-c684d2/auth-test-model-c684d2-kserve-5cb979995c-6xppj"
Oct 30 13:30:38.720692 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:30:38.720570    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08f03e92_3e35_4a93_a240_f38ed6572622.slice/crio-5fbbf91632eb6c5eadd91f9c90d2aab9ca9883878926a127efaa7b1fd683183f WatchSource:0}: Error finding container 5fbbf91632eb6c5eadd91f9c90d2aab9ca9883878926a127efaa7b1fd683183f: Status 404 returned error can't find the container with id 5fbbf91632eb6c5eadd91f9c90d2aab9ca9883878926a127efaa7b1fd683183f
Oct 30 13:30:38.721556 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:38.721527    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-c684d2/auth-test-model-c684d2-kserve-5cb979995c-6xppj"]
Oct 30 13:30:39.440243 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:39.440200    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-c684d2/auth-test-model-c684d2-kserve-5cb979995c-6xppj" event={"ID":"08f03e92-3e35-4a93-a240-f38ed6572622","Type":"ContainerStarted","Data":"4a50304506e90af024cc0c350692c851745cd06d410023a9c746e4f8ea51b25a"}
Oct 30 13:30:39.440449 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:39.440253    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-c684d2/auth-test-model-c684d2-kserve-5cb979995c-6xppj" event={"ID":"08f03e92-3e35-4a93-a240-f38ed6572622","Type":"ContainerStarted","Data":"5fbbf91632eb6c5eadd91f9c90d2aab9ca9883878926a127efaa7b1fd683183f"}
Oct 30 13:30:39.487024 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:39.486962    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-c684d2/auth-test-model-c684d2-kserve-5cb979995c-6xppj" podStartSLOduration=1.486946733 podStartE2EDuration="1.486946733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:39.48391625 +0000 UTC m=+1658.631183491" watchObservedRunningTime="2026-10-30 13:30:39.486946733 +0000 UTC m=+1658.634213973"
Oct 30 13:30:40.400101 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:40.400064    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-c684d2/auth-test-model-c684d2-kserve-5cb979995c-6xppj"
Oct 30 13:30:43.433816 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:43.433785    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f12b80-6c694985fd-g8lxp"
Oct 30 13:30:44.620941 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:44.620902    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f7ee8a-9d8fdd47c-k7slx"]
Oct 30 13:30:44.627453 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:44.627425    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f7ee8a-9d8fdd47c-k7slx"
Oct 30 13:30:44.646456 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:44.646425    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f7ee8a-9d8fdd47c-k7slx"]
Oct 30 13:30:44.656003 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:44.655970    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-f7ee8a-serving-cert\""
Oct 30 13:30:44.739740 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:44.739708    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wv9w\" (UniqueName: \"kubernetes.io/projected/f2075f70-e05f-49de-92f2-aa2bbc2e28e7-kube-api-access-4wv9w\") pod \"maas-api-e2e-shared-a-w1-f7ee8a-9d8fdd47c-k7slx\" (UID: \"f2075f70-e05f-49de-92f2-aa2bbc2e28e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f7ee8a-9d8fdd47c-k7slx"
Oct 30 13:30:44.739914 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:44.739762    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2075f70-e05f-49de-92f2-aa2bbc2e28e7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-f7ee8a-9d8fdd47c-k7slx\" (UID: \"f2075f70-e05f-49de-92f2-aa2bbc2e28e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f7ee8a-9d8fdd47c-k7slx"
Oct 30 13:30:44.739914 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:44.739856    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2075f70-e05f-49de-92f2-aa2bbc2e28e7-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-f7ee8a-9d8fdd47c-k7slx\" (UID: \"f2075f70-e05f-49de-92f2-aa2bbc2e28e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f7ee8a-9d8fdd47c-k7slx"
Oct 30 13:30:44.739914 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:44.739899    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2075f70-e05f-49de-92f2-aa2bbc2e28e7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-f7ee8a-9d8fdd47c-k7slx\" (UID: \"f2075f70-e05f-49de-92f2-aa2bbc2e28e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f7ee8a-9d8fdd47c-k7slx"
Oct 30 13:30:44.840910 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:44.840870    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wv9w\" (UniqueName: \"kubernetes.io/projected/f2075f70-e05f-49de-92f2-aa2bbc2e28e7-kube-api-access-4wv9w\") pod \"maas-api-e2e-shared-a-w1-f7ee8a-9d8fdd47c-k7slx\" (UID: \"f2075f70-e05f-49de-92f2-aa2bbc2e28e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f7ee8a-9d8fdd47c-k7slx"
Oct 30 13:30:44.841118 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:44.840946    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2075f70-e05f-49de-92f2-aa2bbc2e28e7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-f7ee8a-9d8fdd47c-k7slx\" (UID: \"f2075f70-e05f-49de-92f2-aa2bbc2e28e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f7ee8a-9d8fdd47c-k7slx"
Oct 30 13:30:44.841118 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:44.840981    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2075f70-e05f-49de-92f2-aa2bbc2e28e7-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-f7ee8a-9d8fdd47c-k7slx\" (UID: \"f2075f70-e05f-49de-92f2-aa2bbc2e28e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f7ee8a-9d8fdd47c-k7slx"
Oct 30 13:30:44.841280 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:44.841221    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2075f70-e05f-49de-92f2-aa2bbc2e28e7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-f7ee8a-9d8fdd47c-k7slx\" (UID: \"f2075f70-e05f-49de-92f2-aa2bbc2e28e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f7ee8a-9d8fdd47c-k7slx"
Oct 30 13:30:44.842018 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:44.841984    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2075f70-e05f-49de-92f2-aa2bbc2e28e7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-f7ee8a-9d8fdd47c-k7slx\" (UID: \"f2075f70-e05f-49de-92f2-aa2bbc2e28e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f7ee8a-9d8fdd47c-k7slx"
Oct 30 13:30:44.843750 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:44.843722    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2075f70-e05f-49de-92f2-aa2bbc2e28e7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-f7ee8a-9d8fdd47c-k7slx\" (UID: \"f2075f70-e05f-49de-92f2-aa2bbc2e28e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f7ee8a-9d8fdd47c-k7slx"
Oct 30 13:30:44.843859 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:44.843727    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2075f70-e05f-49de-92f2-aa2bbc2e28e7-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-f7ee8a-9d8fdd47c-k7slx\" (UID: \"f2075f70-e05f-49de-92f2-aa2bbc2e28e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f7ee8a-9d8fdd47c-k7slx"
Oct 30 13:30:44.865209 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:44.865123    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wv9w\" (UniqueName: \"kubernetes.io/projected/f2075f70-e05f-49de-92f2-aa2bbc2e28e7-kube-api-access-4wv9w\") pod \"maas-api-e2e-shared-a-w1-f7ee8a-9d8fdd47c-k7slx\" (UID: \"f2075f70-e05f-49de-92f2-aa2bbc2e28e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f7ee8a-9d8fdd47c-k7slx"
Oct 30 13:30:44.947144 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:44.946351    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f7ee8a-9d8fdd47c-k7slx"
Oct 30 13:30:45.307384 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.307349    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f7ee8a-9d8fdd47c-k7slx"]
Oct 30 13:30:45.309348 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:30:45.309321    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2075f70_e05f_49de_92f2_aa2bbc2e28e7.slice/crio-9ed55b57795847390a10146d14f6a0efd0457d336febf04b6a2891198222dde4 WatchSource:0}: Error finding container 9ed55b57795847390a10146d14f6a0efd0457d336febf04b6a2891198222dde4: Status 404 returned error can't find the container with id 9ed55b57795847390a10146d14f6a0efd0457d336febf04b6a2891198222dde4
Oct 30 13:30:45.495161 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.495099    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f7ee8a-9d8fdd47c-k7slx" event={"ID":"f2075f70-e05f-49de-92f2-aa2bbc2e28e7","Type":"ContainerStarted","Data":"9ed55b57795847390a10146d14f6a0efd0457d336febf04b6a2891198222dde4"}
Oct 30 13:30:45.801776 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.801736    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn"]
Oct 30 13:30:45.807798 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.807768    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn"
Oct 30 13:30:45.816708 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.816671    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft"]
Oct 30 13:30:45.824389 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.824356    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft"
Oct 30 13:30:45.836154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.833051    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn"]
Oct 30 13:30:45.867282 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.858367    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-istiod-ca-cert\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft\" (UID: \"aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft"
Oct 30 13:30:45.867282 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.858469    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d087f5d8-cdf1-4014-98ab-8aa0170489c9-workload-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn\" (UID: \"d087f5d8-cdf1-4014-98ab-8aa0170489c9\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn"
Oct 30 13:30:45.867282 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.858511    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-istio-data\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft\" (UID: \"aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft"
Oct 30 13:30:45.867282 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.858537    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-credential-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft\" (UID: \"aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft"
Oct 30 13:30:45.867282 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.858642    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-workload-certs\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft\" (UID: \"aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft"
Oct 30 13:30:45.867282 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.858681    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgvdr\" (UniqueName: \"kubernetes.io/projected/d087f5d8-cdf1-4014-98ab-8aa0170489c9-kube-api-access-pgvdr\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn\" (UID: \"d087f5d8-cdf1-4014-98ab-8aa0170489c9\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn"
Oct 30 13:30:45.867282 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.858705    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw6bs\" (UniqueName: \"kubernetes.io/projected/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-kube-api-access-pw6bs\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft\" (UID: \"aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft"
Oct 30 13:30:45.867282 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.858739    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d087f5d8-cdf1-4014-98ab-8aa0170489c9-istio-data\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn\" (UID: \"d087f5d8-cdf1-4014-98ab-8aa0170489c9\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn"
Oct 30 13:30:45.867282 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.858781    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-istio-envoy\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft\" (UID: \"aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft"
Oct 30 13:30:45.867282 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.858811    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d087f5d8-cdf1-4014-98ab-8aa0170489c9-istio-podinfo\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn\" (UID: \"d087f5d8-cdf1-4014-98ab-8aa0170489c9\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn"
Oct 30 13:30:45.867282 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.858869    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d087f5d8-cdf1-4014-98ab-8aa0170489c9-credential-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn\" (UID: \"d087f5d8-cdf1-4014-98ab-8aa0170489c9\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn"
Oct 30 13:30:45.867282 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.858906    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d087f5d8-cdf1-4014-98ab-8aa0170489c9-istiod-ca-cert\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn\" (UID: \"d087f5d8-cdf1-4014-98ab-8aa0170489c9\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn"
Oct 30 13:30:45.867282 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.858926    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-workload-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft\" (UID: \"aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft"
Oct 30 13:30:45.867282 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.858949    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d087f5d8-cdf1-4014-98ab-8aa0170489c9-workload-certs\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn\" (UID: \"d087f5d8-cdf1-4014-98ab-8aa0170489c9\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn"
Oct 30 13:30:45.867282 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.859014    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d087f5d8-cdf1-4014-98ab-8aa0170489c9-istio-token\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn\" (UID: \"d087f5d8-cdf1-4014-98ab-8aa0170489c9\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn"
Oct 30 13:30:45.867282 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.859067    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-istio-podinfo\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft\" (UID: \"aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft"
Oct 30 13:30:45.867282 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.859094    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d087f5d8-cdf1-4014-98ab-8aa0170489c9-istio-envoy\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn\" (UID: \"d087f5d8-cdf1-4014-98ab-8aa0170489c9\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn"
Oct 30 13:30:45.867282 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.859174    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-istio-token\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft\" (UID: \"aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft"
Oct 30 13:30:45.867282 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.860347    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-f9ebb5-openshift-default-dockercfg-259p4\""
Oct 30 13:30:45.868521 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.867503    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft"]
Oct 30 13:30:45.961291 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.961194    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d087f5d8-cdf1-4014-98ab-8aa0170489c9-workload-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn\" (UID: \"d087f5d8-cdf1-4014-98ab-8aa0170489c9\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn"
Oct 30 13:30:45.961291 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.961275    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-istio-data\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft\" (UID: \"aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft"
Oct 30 13:30:45.961538 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.961308    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-credential-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft\" (UID: \"aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft"
Oct 30 13:30:45.961538 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.961425    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-workload-certs\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft\" (UID: \"aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft"
Oct 30 13:30:45.961538 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.961471    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgvdr\" (UniqueName: \"kubernetes.io/projected/d087f5d8-cdf1-4014-98ab-8aa0170489c9-kube-api-access-pgvdr\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn\" (UID: \"d087f5d8-cdf1-4014-98ab-8aa0170489c9\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn"
Oct 30 13:30:45.961538 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.961504    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw6bs\" (UniqueName: \"kubernetes.io/projected/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-kube-api-access-pw6bs\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft\" (UID: \"aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft"
Oct 30 13:30:45.961758 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.961563    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d087f5d8-cdf1-4014-98ab-8aa0170489c9-istio-data\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn\" (UID: \"d087f5d8-cdf1-4014-98ab-8aa0170489c9\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn"
Oct 30 13:30:45.961758 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.961625    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-istio-envoy\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft\" (UID: \"aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft"
Oct 30 13:30:45.961758 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.961657    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d087f5d8-cdf1-4014-98ab-8aa0170489c9-istio-podinfo\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn\" (UID: \"d087f5d8-cdf1-4014-98ab-8aa0170489c9\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn"
Oct 30 13:30:45.961758 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.961658    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d087f5d8-cdf1-4014-98ab-8aa0170489c9-workload-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn\" (UID: \"d087f5d8-cdf1-4014-98ab-8aa0170489c9\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn"
Oct 30 13:30:45.961758 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.961734    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-istio-data\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft\" (UID: \"aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft"
Oct 30 13:30:45.961758 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.961749    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d087f5d8-cdf1-4014-98ab-8aa0170489c9-credential-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn\" (UID: \"d087f5d8-cdf1-4014-98ab-8aa0170489c9\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn"
Oct 30 13:30:45.962072 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.961768    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-credential-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft\" (UID: \"aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft"
Oct 30 13:30:45.962072 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.961802    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d087f5d8-cdf1-4014-98ab-8aa0170489c9-istiod-ca-cert\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn\" (UID: \"d087f5d8-cdf1-4014-98ab-8aa0170489c9\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn"
Oct 30 13:30:45.962072 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.961827    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-workload-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft\" (UID: \"aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft"
Oct 30 13:30:45.962072 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.961856    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d087f5d8-cdf1-4014-98ab-8aa0170489c9-workload-certs\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn\" (UID: \"d087f5d8-cdf1-4014-98ab-8aa0170489c9\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn"
Oct 30 13:30:45.962072 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.962026    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d087f5d8-cdf1-4014-98ab-8aa0170489c9-istio-data\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn\" (UID: \"d087f5d8-cdf1-4014-98ab-8aa0170489c9\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn"
Oct 30 13:30:45.963546 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.962420    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-workload-certs\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft\" (UID: \"aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft"
Oct 30 13:30:45.963546 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.962448    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-workload-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft\" (UID: \"aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft"
Oct 30 13:30:45.963546 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.962474    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d087f5d8-cdf1-4014-98ab-8aa0170489c9-istio-token\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn\" (UID: \"d087f5d8-cdf1-4014-98ab-8aa0170489c9\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn"
Oct 30 13:30:45.963546 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.962496    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d087f5d8-cdf1-4014-98ab-8aa0170489c9-credential-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn\" (UID: \"d087f5d8-cdf1-4014-98ab-8aa0170489c9\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn"
Oct 30 13:30:45.963546 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.962525    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-istio-podinfo\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft\" (UID: \"aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft"
Oct 30 13:30:45.963546 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.962559    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d087f5d8-cdf1-4014-98ab-8aa0170489c9-istio-envoy\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn\" (UID: \"d087f5d8-cdf1-4014-98ab-8aa0170489c9\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn"
Oct 30 13:30:45.963546 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.962598    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-istio-token\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft\" (UID: \"aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft"
Oct 30 13:30:45.963546 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.962670    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-istiod-ca-cert\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft\" (UID: \"aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft"
Oct 30 13:30:45.963546 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.963481    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-istiod-ca-cert\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft\" (UID: \"aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft"
Oct 30 13:30:45.963546 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.963497    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d087f5d8-cdf1-4014-98ab-8aa0170489c9-istiod-ca-cert\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn\" (UID: \"d087f5d8-cdf1-4014-98ab-8aa0170489c9\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn"
Oct 30 13:30:45.964462 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.964435    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d087f5d8-cdf1-4014-98ab-8aa0170489c9-workload-certs\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn\" (UID: \"d087f5d8-cdf1-4014-98ab-8aa0170489c9\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn"
Oct 30 13:30:45.964619 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.964595    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-istio-envoy\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft\" (UID: \"aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft"
Oct 30 13:30:45.964816 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.964793    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d087f5d8-cdf1-4014-98ab-8aa0170489c9-istio-podinfo\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn\" (UID: \"d087f5d8-cdf1-4014-98ab-8aa0170489c9\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn"
Oct 30 13:30:45.965049 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.965033    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d087f5d8-cdf1-4014-98ab-8aa0170489c9-istio-envoy\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn\" (UID: \"d087f5d8-cdf1-4014-98ab-8aa0170489c9\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn"
Oct 30 13:30:45.965695 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.965668    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-istio-podinfo\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft\" (UID: \"aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft"
Oct 30 13:30:45.983806 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.983774    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-istio-token\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft\" (UID: \"aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft"
Oct 30 13:30:45.983954 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.983772    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d087f5d8-cdf1-4014-98ab-8aa0170489c9-istio-token\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn\" (UID: \"d087f5d8-cdf1-4014-98ab-8aa0170489c9\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn"
Oct 30 13:30:45.983954 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.983834    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw6bs\" (UniqueName: \"kubernetes.io/projected/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-kube-api-access-pw6bs\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft\" (UID: \"aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft"
Oct 30 13:30:45.984043 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:45.983980    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgvdr\" (UniqueName: \"kubernetes.io/projected/d087f5d8-cdf1-4014-98ab-8aa0170489c9-kube-api-access-pgvdr\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn\" (UID: \"d087f5d8-cdf1-4014-98ab-8aa0170489c9\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn"
Oct 30 13:30:46.178389 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:46.178349    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn"
Oct 30 13:30:46.191147 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:46.191105    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft"
Oct 30 13:30:46.536041 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:46.535905    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f7ee8a-9d8fdd47c-k7slx" event={"ID":"f2075f70-e05f-49de-92f2-aa2bbc2e28e7","Type":"ContainerStarted","Data":"cffe0abaa9ef85e4872ca542f099ee95b0f2d2e19e75b8b0fb8167d2ef1470da"}
Oct 30 13:30:46.537392 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:46.537363    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f7ee8a-9d8fdd47c-k7slx"
Oct 30 13:30:46.566193 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:30:46.566160    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd087f5d8_cdf1_4014_98ab_8aa0170489c9.slice/crio-c195c243fbe29330898a8c733183a3379976154b42201867ae01aa2ccedd57e6 WatchSource:0}: Error finding container c195c243fbe29330898a8c733183a3379976154b42201867ae01aa2ccedd57e6: Status 404 returned error can't find the container with id c195c243fbe29330898a8c733183a3379976154b42201867ae01aa2ccedd57e6
Oct 30 13:30:46.568344 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:46.568316    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn"]
Oct 30 13:30:46.569164 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:46.569106    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:30:46.569247 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:46.569194    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:30:46.569247 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:46.569229    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:30:46.601672 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:46.601610    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f7ee8a-9d8fdd47c-k7slx" podStartSLOduration=2.23654228 podStartE2EDuration="2.601590324s" totalImagesPullingTime="365.048044ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:46.585735467 +0000 UTC m=+1665.733002710" watchObservedRunningTime="2026-10-30 13:30:46.601590324 +0000 UTC m=+1665.748857565"
Oct 30 13:30:46.608166 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:46.608101    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft"]
Oct 30 13:30:46.613544 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:30:46.613515    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaefd92b_7e59_4d40_9b4f_d6b6a21d97e8.slice/crio-4637e237d0f10102570ac15499752e872b1d628112582d8cedb157cb2e58b572 WatchSource:0}: Error finding container 4637e237d0f10102570ac15499752e872b1d628112582d8cedb157cb2e58b572: Status 404 returned error can't find the container with id 4637e237d0f10102570ac15499752e872b1d628112582d8cedb157cb2e58b572
Oct 30 13:30:46.617180 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:46.617109    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:30:46.617281 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:46.617234    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:30:46.617351 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:46.617278    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:30:47.639157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:47.625316    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft" event={"ID":"aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8","Type":"ContainerStarted","Data":"78d7dc53d6bd6c760b98468350d1fa315f64499ada0c47bd00ab8a24011f31c8"}
Oct 30 13:30:47.639157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:47.625387    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft" event={"ID":"aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8","Type":"ContainerStarted","Data":"4637e237d0f10102570ac15499752e872b1d628112582d8cedb157cb2e58b572"}
Oct 30 13:30:47.639157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:47.629423    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn" event={"ID":"d087f5d8-cdf1-4014-98ab-8aa0170489c9","Type":"ContainerStarted","Data":"199faf04f21057ea20fb54422b914fb39c18061608a7b29b189a31e311d460f8"}
Oct 30 13:30:47.639157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:47.629472    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn" event={"ID":"d087f5d8-cdf1-4014-98ab-8aa0170489c9","Type":"ContainerStarted","Data":"c195c243fbe29330898a8c733183a3379976154b42201867ae01aa2ccedd57e6"}
Oct 30 13:30:47.692157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:47.691100    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft" podStartSLOduration=2.691078036 podStartE2EDuration="2.691078036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:47.690282287 +0000 UTC m=+1666.837549529" watchObservedRunningTime="2026-10-30 13:30:47.691078036 +0000 UTC m=+1666.838345304"
Oct 30 13:30:48.187174 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:48.183253    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn"
Oct 30 13:30:48.187393 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:48.187234    5093 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Oct 30 13:30:48.187393 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:48.187289    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn" podUID="d087f5d8-cdf1-4014-98ab-8aa0170489c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Oct 30 13:30:48.192211 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:48.192183    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft"
Oct 30 13:30:48.192990 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:48.192969    5093 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Oct 30 13:30:48.193064 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:48.193013    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft" podUID="aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Oct 30 13:30:48.400237 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:48.400180    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-c684d2/auth-test-model-c684d2-kserve-5cb979995c-6xppj"
Oct 30 13:30:48.413729 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:48.413700    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-c684d2/auth-test-model-c684d2-kserve-5cb979995c-6xppj"
Oct 30 13:30:48.490251 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:48.488381    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn" podStartSLOduration=3.48835963 podStartE2EDuration="3.48835963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:47.756528498 +0000 UTC m=+1666.903795740" watchObservedRunningTime="2026-10-30 13:30:48.48835963 +0000 UTC m=+1667.635626872"
Oct 30 13:30:48.701959 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:48.701804    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-c684d2/auth-test-model-c684d2-kserve-5cb979995c-6xppj"
Oct 30 13:30:49.184158 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:49.180686    5093 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Oct 30 13:30:49.184158 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:49.180795    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn" podUID="d087f5d8-cdf1-4014-98ab-8aa0170489c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Oct 30 13:30:49.191690 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:49.191643    5093 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Oct 30 13:30:49.191897 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:49.191702    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft" podUID="aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Oct 30 13:30:50.185345 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:50.185304    5093 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Oct 30 13:30:50.185994 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:50.185944    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn" podUID="d087f5d8-cdf1-4014-98ab-8aa0170489c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Oct 30 13:30:50.191823 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:50.191660    5093 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Oct 30 13:30:50.191823 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:50.191728    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft" podUID="aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Oct 30 13:30:51.181153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:51.180605    5093 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Oct 30 13:30:51.181153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:51.180668    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn" podUID="d087f5d8-cdf1-4014-98ab-8aa0170489c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Oct 30 13:30:51.191537 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:51.191494    5093 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Oct 30 13:30:51.192061 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:51.191556    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft" podUID="aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Oct 30 13:30:51.202713 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:30:51.202654    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:30:52.187219 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:52.182681    5093 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Oct 30 13:30:52.187219 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:52.182767    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn" podUID="d087f5d8-cdf1-4014-98ab-8aa0170489c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Oct 30 13:30:52.191600 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:52.191562    5093 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Oct 30 13:30:52.191965 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:52.191627    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft" podUID="aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Oct 30 13:30:53.179832 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:53.179784    5093 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Oct 30 13:30:53.180003 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:53.179872    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn" podUID="d087f5d8-cdf1-4014-98ab-8aa0170489c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Oct 30 13:30:53.191556 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:53.191518    5093 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Oct 30 13:30:53.191694 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:53.191597    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft" podUID="aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Oct 30 13:30:53.691847 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:53.691817    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f7ee8a-9d8fdd47c-k7slx"
Oct 30 13:30:54.429106 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:54.429072    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn"
Oct 30 13:30:54.431013 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:54.430989    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn"
Oct 30 13:30:54.433487 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:54.433467    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn"
Oct 30 13:30:54.440481 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:54.440457    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft"
Oct 30 13:30:54.441889 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:54.441866    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft"
Oct 30 13:30:54.442816 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:30:54.442794    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft"
Oct 30 13:31:04.928970 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:04.928937    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-22af91/auth-test-model-22af91-kserve-84f6df67c7-hxtzx"]
Oct 30 13:31:04.934828 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:04.934802    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-22af91/auth-test-model-22af91-kserve-84f6df67c7-hxtzx"
Oct 30 13:31:04.960254 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:04.955458    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-22af91/auth-test-model-22af91-kserve-84f6df67c7-hxtzx"]
Oct 30 13:31:04.968835 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:04.968804    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-22af91\"/\"default-dockercfg-kcgpd\""
Oct 30 13:31:04.969016 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:04.968926    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-22af91\"/\"auth-test-model-22af91-kserve-self-signed-certs\""
Oct 30 13:31:04.969016 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:04.969014    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-22af91\"/\"kube-root-ca.crt\""
Oct 30 13:31:04.969262 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:04.969089    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-22af91\"/\"openshift-service-ca.crt\""
Oct 30 13:31:05.082454 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:05.082404    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21d2769d-dca2-4944-b72c-c18dc3a1e99d-tmp-dir\") pod \"auth-test-model-22af91-kserve-84f6df67c7-hxtzx\" (UID: \"21d2769d-dca2-4944-b72c-c18dc3a1e99d\") " pod="ai-tenant-e2e-shared-b-w2-22af91/auth-test-model-22af91-kserve-84f6df67c7-hxtzx"
Oct 30 13:31:05.082454 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:05.082460    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21d2769d-dca2-4944-b72c-c18dc3a1e99d-tls-certs\") pod \"auth-test-model-22af91-kserve-84f6df67c7-hxtzx\" (UID: \"21d2769d-dca2-4944-b72c-c18dc3a1e99d\") " pod="ai-tenant-e2e-shared-b-w2-22af91/auth-test-model-22af91-kserve-84f6df67c7-hxtzx"
Oct 30 13:31:05.082673 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:05.082589    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqj5g\" (UniqueName: \"kubernetes.io/projected/21d2769d-dca2-4944-b72c-c18dc3a1e99d-kube-api-access-hqj5g\") pod \"auth-test-model-22af91-kserve-84f6df67c7-hxtzx\" (UID: \"21d2769d-dca2-4944-b72c-c18dc3a1e99d\") " pod="ai-tenant-e2e-shared-b-w2-22af91/auth-test-model-22af91-kserve-84f6df67c7-hxtzx"
Oct 30 13:31:05.082673 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:05.082641    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21d2769d-dca2-4944-b72c-c18dc3a1e99d-dshm\") pod \"auth-test-model-22af91-kserve-84f6df67c7-hxtzx\" (UID: \"21d2769d-dca2-4944-b72c-c18dc3a1e99d\") " pod="ai-tenant-e2e-shared-b-w2-22af91/auth-test-model-22af91-kserve-84f6df67c7-hxtzx"
Oct 30 13:31:05.082839 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:05.082818    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21d2769d-dca2-4944-b72c-c18dc3a1e99d-model-cache\") pod \"auth-test-model-22af91-kserve-84f6df67c7-hxtzx\" (UID: \"21d2769d-dca2-4944-b72c-c18dc3a1e99d\") " pod="ai-tenant-e2e-shared-b-w2-22af91/auth-test-model-22af91-kserve-84f6df67c7-hxtzx"
Oct 30 13:31:05.082900 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:05.082866    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21d2769d-dca2-4944-b72c-c18dc3a1e99d-home\") pod \"auth-test-model-22af91-kserve-84f6df67c7-hxtzx\" (UID: \"21d2769d-dca2-4944-b72c-c18dc3a1e99d\") " pod="ai-tenant-e2e-shared-b-w2-22af91/auth-test-model-22af91-kserve-84f6df67c7-hxtzx"
Oct 30 13:31:05.194216 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:05.194057    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21d2769d-dca2-4944-b72c-c18dc3a1e99d-tmp-dir\") pod \"auth-test-model-22af91-kserve-84f6df67c7-hxtzx\" (UID: \"21d2769d-dca2-4944-b72c-c18dc3a1e99d\") " pod="ai-tenant-e2e-shared-b-w2-22af91/auth-test-model-22af91-kserve-84f6df67c7-hxtzx"
Oct 30 13:31:05.194216 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:05.194111    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21d2769d-dca2-4944-b72c-c18dc3a1e99d-tls-certs\") pod \"auth-test-model-22af91-kserve-84f6df67c7-hxtzx\" (UID: \"21d2769d-dca2-4944-b72c-c18dc3a1e99d\") " pod="ai-tenant-e2e-shared-b-w2-22af91/auth-test-model-22af91-kserve-84f6df67c7-hxtzx"
Oct 30 13:31:05.194457 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:05.194220    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqj5g\" (UniqueName: \"kubernetes.io/projected/21d2769d-dca2-4944-b72c-c18dc3a1e99d-kube-api-access-hqj5g\") pod \"auth-test-model-22af91-kserve-84f6df67c7-hxtzx\" (UID: \"21d2769d-dca2-4944-b72c-c18dc3a1e99d\") " pod="ai-tenant-e2e-shared-b-w2-22af91/auth-test-model-22af91-kserve-84f6df67c7-hxtzx"
Oct 30 13:31:05.194457 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:05.194278    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21d2769d-dca2-4944-b72c-c18dc3a1e99d-dshm\") pod \"auth-test-model-22af91-kserve-84f6df67c7-hxtzx\" (UID: \"21d2769d-dca2-4944-b72c-c18dc3a1e99d\") " pod="ai-tenant-e2e-shared-b-w2-22af91/auth-test-model-22af91-kserve-84f6df67c7-hxtzx"
Oct 30 13:31:05.194457 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:05.194386    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21d2769d-dca2-4944-b72c-c18dc3a1e99d-model-cache\") pod \"auth-test-model-22af91-kserve-84f6df67c7-hxtzx\" (UID: \"21d2769d-dca2-4944-b72c-c18dc3a1e99d\") " pod="ai-tenant-e2e-shared-b-w2-22af91/auth-test-model-22af91-kserve-84f6df67c7-hxtzx"
Oct 30 13:31:05.194457 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:05.194424    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21d2769d-dca2-4944-b72c-c18dc3a1e99d-home\") pod \"auth-test-model-22af91-kserve-84f6df67c7-hxtzx\" (UID: \"21d2769d-dca2-4944-b72c-c18dc3a1e99d\") " pod="ai-tenant-e2e-shared-b-w2-22af91/auth-test-model-22af91-kserve-84f6df67c7-hxtzx"
Oct 30 13:31:05.195194 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:05.195167    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21d2769d-dca2-4944-b72c-c18dc3a1e99d-tmp-dir\") pod \"auth-test-model-22af91-kserve-84f6df67c7-hxtzx\" (UID: \"21d2769d-dca2-4944-b72c-c18dc3a1e99d\") " pod="ai-tenant-e2e-shared-b-w2-22af91/auth-test-model-22af91-kserve-84f6df67c7-hxtzx"
Oct 30 13:31:05.195931 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:05.195888    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21d2769d-dca2-4944-b72c-c18dc3a1e99d-model-cache\") pod \"auth-test-model-22af91-kserve-84f6df67c7-hxtzx\" (UID: \"21d2769d-dca2-4944-b72c-c18dc3a1e99d\") " pod="ai-tenant-e2e-shared-b-w2-22af91/auth-test-model-22af91-kserve-84f6df67c7-hxtzx"
Oct 30 13:31:05.196200 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:05.196179    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21d2769d-dca2-4944-b72c-c18dc3a1e99d-home\") pod \"auth-test-model-22af91-kserve-84f6df67c7-hxtzx\" (UID: \"21d2769d-dca2-4944-b72c-c18dc3a1e99d\") " pod="ai-tenant-e2e-shared-b-w2-22af91/auth-test-model-22af91-kserve-84f6df67c7-hxtzx"
Oct 30 13:31:05.220339 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:05.220296    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21d2769d-dca2-4944-b72c-c18dc3a1e99d-dshm\") pod \"auth-test-model-22af91-kserve-84f6df67c7-hxtzx\" (UID: \"21d2769d-dca2-4944-b72c-c18dc3a1e99d\") " pod="ai-tenant-e2e-shared-b-w2-22af91/auth-test-model-22af91-kserve-84f6df67c7-hxtzx"
Oct 30 13:31:05.228160 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:05.224583    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqj5g\" (UniqueName: \"kubernetes.io/projected/21d2769d-dca2-4944-b72c-c18dc3a1e99d-kube-api-access-hqj5g\") pod \"auth-test-model-22af91-kserve-84f6df67c7-hxtzx\" (UID: \"21d2769d-dca2-4944-b72c-c18dc3a1e99d\") " pod="ai-tenant-e2e-shared-b-w2-22af91/auth-test-model-22af91-kserve-84f6df67c7-hxtzx"
Oct 30 13:31:05.228160 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:05.225421    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21d2769d-dca2-4944-b72c-c18dc3a1e99d-tls-certs\") pod \"auth-test-model-22af91-kserve-84f6df67c7-hxtzx\" (UID: \"21d2769d-dca2-4944-b72c-c18dc3a1e99d\") " pod="ai-tenant-e2e-shared-b-w2-22af91/auth-test-model-22af91-kserve-84f6df67c7-hxtzx"
Oct 30 13:31:05.276794 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:05.276759    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-22af91/auth-test-model-22af91-kserve-84f6df67c7-hxtzx"
Oct 30 13:31:05.609707 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:05.609669    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-22af91/auth-test-model-22af91-kserve-84f6df67c7-hxtzx"]
Oct 30 13:31:05.610315 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:31:05.610288    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21d2769d_dca2_4944_b72c_c18dc3a1e99d.slice/crio-422abcc1c99cc9ec5b0ef61861cda9342eb98fb0f1682eb73fdccb6d18be61d8 WatchSource:0}: Error finding container 422abcc1c99cc9ec5b0ef61861cda9342eb98fb0f1682eb73fdccb6d18be61d8: Status 404 returned error can't find the container with id 422abcc1c99cc9ec5b0ef61861cda9342eb98fb0f1682eb73fdccb6d18be61d8
Oct 30 13:31:05.987088 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:05.987041    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-22af91/auth-test-model-22af91-kserve-84f6df67c7-hxtzx" event={"ID":"21d2769d-dca2-4944-b72c-c18dc3a1e99d","Type":"ContainerStarted","Data":"6feb364171de4f926887652a4d5ee2b0397cb08fc6010ce5d856093d675bc327"}
Oct 30 13:31:05.987088 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:05.987094    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-22af91/auth-test-model-22af91-kserve-84f6df67c7-hxtzx" event={"ID":"21d2769d-dca2-4944-b72c-c18dc3a1e99d","Type":"ContainerStarted","Data":"422abcc1c99cc9ec5b0ef61861cda9342eb98fb0f1682eb73fdccb6d18be61d8"}
Oct 30 13:31:06.032736 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:06.032677    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-22af91/auth-test-model-22af91-kserve-84f6df67c7-hxtzx" podStartSLOduration=2.032661145 podStartE2EDuration="2.032661145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:31:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:31:06.028722604 +0000 UTC m=+1685.175989847" watchObservedRunningTime="2026-10-30 13:31:06.032661145 +0000 UTC m=+1685.179928426"
Oct 30 13:31:06.277399 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:06.277294    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-22af91/auth-test-model-22af91-kserve-84f6df67c7-hxtzx"
Oct 30 13:31:15.277756 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:15.277714    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-22af91/auth-test-model-22af91-kserve-84f6df67c7-hxtzx"
Oct 30 13:31:15.291666 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:15.291618    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-22af91/auth-test-model-22af91-kserve-84f6df67c7-hxtzx"
Oct 30 13:31:16.096895 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:16.096863    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-22af91/auth-test-model-22af91-kserve-84f6df67c7-hxtzx"
Oct 30 13:31:21.171499 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:31:21.171450    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:31:34.083477 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:34.083437    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-f45f04/autores-model-f45f04-kserve-854dc88d7c-lt9q8"]
Oct 30 13:31:34.090390 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:34.090363    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-f45f04/autores-model-f45f04-kserve-854dc88d7c-lt9q8"
Oct 30 13:31:34.110384 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:34.110347    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-f45f04/autores-model-f45f04-kserve-854dc88d7c-lt9q8"]
Oct 30 13:31:34.114753 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:34.114726    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-f45f04\"/\"autores-model-f45f04-kserve-self-signed-certs\""
Oct 30 13:31:34.114753 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:34.114726    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-f45f04\"/\"default-dockercfg-tcsq4\""
Oct 30 13:31:34.119094 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:34.119058    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-f45f04\"/\"kube-root-ca.crt\""
Oct 30 13:31:34.119094 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:34.119100    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-f45f04\"/\"openshift-service-ca.crt\""
Oct 30 13:31:34.197907 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:34.197869    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5f4950f-ab65-4763-9d96-e4210dd6e955-dshm\") pod \"autores-model-f45f04-kserve-854dc88d7c-lt9q8\" (UID: \"a5f4950f-ab65-4763-9d96-e4210dd6e955\") " pod="ai-tenant-e2e-autores-f45f04/autores-model-f45f04-kserve-854dc88d7c-lt9q8"
Oct 30 13:31:34.198097 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:34.197935    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5f4950f-ab65-4763-9d96-e4210dd6e955-tmp-dir\") pod \"autores-model-f45f04-kserve-854dc88d7c-lt9q8\" (UID: \"a5f4950f-ab65-4763-9d96-e4210dd6e955\") " pod="ai-tenant-e2e-autores-f45f04/autores-model-f45f04-kserve-854dc88d7c-lt9q8"
Oct 30 13:31:34.198217 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:34.198088    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5f4950f-ab65-4763-9d96-e4210dd6e955-home\") pod \"autores-model-f45f04-kserve-854dc88d7c-lt9q8\" (UID: \"a5f4950f-ab65-4763-9d96-e4210dd6e955\") " pod="ai-tenant-e2e-autores-f45f04/autores-model-f45f04-kserve-854dc88d7c-lt9q8"
Oct 30 13:31:34.198217 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:34.198154    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5f4950f-ab65-4763-9d96-e4210dd6e955-model-cache\") pod \"autores-model-f45f04-kserve-854dc88d7c-lt9q8\" (UID: \"a5f4950f-ab65-4763-9d96-e4210dd6e955\") " pod="ai-tenant-e2e-autores-f45f04/autores-model-f45f04-kserve-854dc88d7c-lt9q8"
Oct 30 13:31:34.198298 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:34.198262    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5f4950f-ab65-4763-9d96-e4210dd6e955-tls-certs\") pod \"autores-model-f45f04-kserve-854dc88d7c-lt9q8\" (UID: \"a5f4950f-ab65-4763-9d96-e4210dd6e955\") " pod="ai-tenant-e2e-autores-f45f04/autores-model-f45f04-kserve-854dc88d7c-lt9q8"
Oct 30 13:31:34.198338 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:34.198299    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-787rc\" (UniqueName: \"kubernetes.io/projected/a5f4950f-ab65-4763-9d96-e4210dd6e955-kube-api-access-787rc\") pod \"autores-model-f45f04-kserve-854dc88d7c-lt9q8\" (UID: \"a5f4950f-ab65-4763-9d96-e4210dd6e955\") " pod="ai-tenant-e2e-autores-f45f04/autores-model-f45f04-kserve-854dc88d7c-lt9q8"
Oct 30 13:31:34.299505 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:34.299453    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5f4950f-ab65-4763-9d96-e4210dd6e955-tmp-dir\") pod \"autores-model-f45f04-kserve-854dc88d7c-lt9q8\" (UID: \"a5f4950f-ab65-4763-9d96-e4210dd6e955\") " pod="ai-tenant-e2e-autores-f45f04/autores-model-f45f04-kserve-854dc88d7c-lt9q8"
Oct 30 13:31:34.299719 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:34.299597    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5f4950f-ab65-4763-9d96-e4210dd6e955-home\") pod \"autores-model-f45f04-kserve-854dc88d7c-lt9q8\" (UID: \"a5f4950f-ab65-4763-9d96-e4210dd6e955\") " pod="ai-tenant-e2e-autores-f45f04/autores-model-f45f04-kserve-854dc88d7c-lt9q8"
Oct 30 13:31:34.299719 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:34.299652    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5f4950f-ab65-4763-9d96-e4210dd6e955-model-cache\") pod \"autores-model-f45f04-kserve-854dc88d7c-lt9q8\" (UID: \"a5f4950f-ab65-4763-9d96-e4210dd6e955\") " pod="ai-tenant-e2e-autores-f45f04/autores-model-f45f04-kserve-854dc88d7c-lt9q8"
Oct 30 13:31:34.299847 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:34.299746    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5f4950f-ab65-4763-9d96-e4210dd6e955-tls-certs\") pod \"autores-model-f45f04-kserve-854dc88d7c-lt9q8\" (UID: \"a5f4950f-ab65-4763-9d96-e4210dd6e955\") " pod="ai-tenant-e2e-autores-f45f04/autores-model-f45f04-kserve-854dc88d7c-lt9q8"
Oct 30 13:31:34.299847 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:34.299826    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-787rc\" (UniqueName: \"kubernetes.io/projected/a5f4950f-ab65-4763-9d96-e4210dd6e955-kube-api-access-787rc\") pod \"autores-model-f45f04-kserve-854dc88d7c-lt9q8\" (UID: \"a5f4950f-ab65-4763-9d96-e4210dd6e955\") " pod="ai-tenant-e2e-autores-f45f04/autores-model-f45f04-kserve-854dc88d7c-lt9q8"
Oct 30 13:31:34.299957 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:34.299939    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5f4950f-ab65-4763-9d96-e4210dd6e955-dshm\") pod \"autores-model-f45f04-kserve-854dc88d7c-lt9q8\" (UID: \"a5f4950f-ab65-4763-9d96-e4210dd6e955\") " pod="ai-tenant-e2e-autores-f45f04/autores-model-f45f04-kserve-854dc88d7c-lt9q8"
Oct 30 13:31:34.300488 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:34.300459    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5f4950f-ab65-4763-9d96-e4210dd6e955-tmp-dir\") pod \"autores-model-f45f04-kserve-854dc88d7c-lt9q8\" (UID: \"a5f4950f-ab65-4763-9d96-e4210dd6e955\") " pod="ai-tenant-e2e-autores-f45f04/autores-model-f45f04-kserve-854dc88d7c-lt9q8"
Oct 30 13:31:34.300740 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:34.300713    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5f4950f-ab65-4763-9d96-e4210dd6e955-home\") pod \"autores-model-f45f04-kserve-854dc88d7c-lt9q8\" (UID: \"a5f4950f-ab65-4763-9d96-e4210dd6e955\") " pod="ai-tenant-e2e-autores-f45f04/autores-model-f45f04-kserve-854dc88d7c-lt9q8"
Oct 30 13:31:34.300831 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:34.300724    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5f4950f-ab65-4763-9d96-e4210dd6e955-model-cache\") pod \"autores-model-f45f04-kserve-854dc88d7c-lt9q8\" (UID: \"a5f4950f-ab65-4763-9d96-e4210dd6e955\") " pod="ai-tenant-e2e-autores-f45f04/autores-model-f45f04-kserve-854dc88d7c-lt9q8"
Oct 30 13:31:34.302797 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:34.302763    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5f4950f-ab65-4763-9d96-e4210dd6e955-dshm\") pod \"autores-model-f45f04-kserve-854dc88d7c-lt9q8\" (UID: \"a5f4950f-ab65-4763-9d96-e4210dd6e955\") " pod="ai-tenant-e2e-autores-f45f04/autores-model-f45f04-kserve-854dc88d7c-lt9q8"
Oct 30 13:31:34.303798 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:34.303774    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5f4950f-ab65-4763-9d96-e4210dd6e955-tls-certs\") pod \"autores-model-f45f04-kserve-854dc88d7c-lt9q8\" (UID: \"a5f4950f-ab65-4763-9d96-e4210dd6e955\") " pod="ai-tenant-e2e-autores-f45f04/autores-model-f45f04-kserve-854dc88d7c-lt9q8"
Oct 30 13:31:34.321040 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:34.321009    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-787rc\" (UniqueName: \"kubernetes.io/projected/a5f4950f-ab65-4763-9d96-e4210dd6e955-kube-api-access-787rc\") pod \"autores-model-f45f04-kserve-854dc88d7c-lt9q8\" (UID: \"a5f4950f-ab65-4763-9d96-e4210dd6e955\") " pod="ai-tenant-e2e-autores-f45f04/autores-model-f45f04-kserve-854dc88d7c-lt9q8"
Oct 30 13:31:34.410646 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:34.410556    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-f45f04/autores-model-f45f04-kserve-854dc88d7c-lt9q8"
Oct 30 13:31:34.743047 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:31:34.743014    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5f4950f_ab65_4763_9d96_e4210dd6e955.slice/crio-56c7526c5fd614d46b8897c26b035d67b6621db2391c3d723a91ca22a744abef WatchSource:0}: Error finding container 56c7526c5fd614d46b8897c26b035d67b6621db2391c3d723a91ca22a744abef: Status 404 returned error can't find the container with id 56c7526c5fd614d46b8897c26b035d67b6621db2391c3d723a91ca22a744abef
Oct 30 13:31:34.743270 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:34.743237    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-f45f04/autores-model-f45f04-kserve-854dc88d7c-lt9q8"]
Oct 30 13:31:35.255250 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:35.255212    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-f45f04/autores-model-f45f04-kserve-854dc88d7c-lt9q8" event={"ID":"a5f4950f-ab65-4763-9d96-e4210dd6e955","Type":"ContainerStarted","Data":"2c90a419c4306dc73ced146ea2d4297eff824844f56d2ad4c46747dce16620f9"}
Oct 30 13:31:35.255250 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:35.255253    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-f45f04/autores-model-f45f04-kserve-854dc88d7c-lt9q8" event={"ID":"a5f4950f-ab65-4763-9d96-e4210dd6e955","Type":"ContainerStarted","Data":"56c7526c5fd614d46b8897c26b035d67b6621db2391c3d723a91ca22a744abef"}
Oct 30 13:31:35.301512 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:35.301453    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-f45f04/autores-model-f45f04-kserve-854dc88d7c-lt9q8" podStartSLOduration=1.301437622 podStartE2EDuration="1.301437622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:31:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:31:35.298376943 +0000 UTC m=+1714.445644196" watchObservedRunningTime="2026-10-30 13:31:35.301437622 +0000 UTC m=+1714.448704888"
Oct 30 13:31:35.412497 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:35.412208    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-f45f04/autores-model-f45f04-kserve-854dc88d7c-lt9q8"
Oct 30 13:31:43.327553 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:43.327513    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8"]
Oct 30 13:31:43.334471 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:43.334445    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8"
Oct 30 13:31:43.354371 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:43.354336    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8"]
Oct 30 13:31:43.362485 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:43.362428    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-22af91-openshift-default-dockercfg-g9dfk\""
Oct 30 13:31:43.486322 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:43.486285    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/428b29c0-c277-4e39-ab76-10655b7382ba-istio-envoy\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8\" (UID: \"428b29c0-c277-4e39-ab76-10655b7382ba\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8"
Oct 30 13:31:43.486500 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:43.486332    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmmrl\" (UniqueName: \"kubernetes.io/projected/428b29c0-c277-4e39-ab76-10655b7382ba-kube-api-access-vmmrl\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8\" (UID: \"428b29c0-c277-4e39-ab76-10655b7382ba\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8"
Oct 30 13:31:43.486500 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:43.486365    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/428b29c0-c277-4e39-ab76-10655b7382ba-workload-socket\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8\" (UID: \"428b29c0-c277-4e39-ab76-10655b7382ba\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8"
Oct 30 13:31:43.486500 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:43.486435    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/428b29c0-c277-4e39-ab76-10655b7382ba-istio-token\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8\" (UID: \"428b29c0-c277-4e39-ab76-10655b7382ba\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8"
Oct 30 13:31:43.486500 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:43.486464    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/428b29c0-c277-4e39-ab76-10655b7382ba-workload-certs\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8\" (UID: \"428b29c0-c277-4e39-ab76-10655b7382ba\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8"
Oct 30 13:31:43.486500 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:43.486498    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/428b29c0-c277-4e39-ab76-10655b7382ba-istio-podinfo\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8\" (UID: \"428b29c0-c277-4e39-ab76-10655b7382ba\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8"
Oct 30 13:31:43.486709 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:43.486582    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/428b29c0-c277-4e39-ab76-10655b7382ba-credential-socket\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8\" (UID: \"428b29c0-c277-4e39-ab76-10655b7382ba\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8"
Oct 30 13:31:43.486709 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:43.486646    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/428b29c0-c277-4e39-ab76-10655b7382ba-istio-data\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8\" (UID: \"428b29c0-c277-4e39-ab76-10655b7382ba\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8"
Oct 30 13:31:43.486785 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:43.486734    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/428b29c0-c277-4e39-ab76-10655b7382ba-istiod-ca-cert\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8\" (UID: \"428b29c0-c277-4e39-ab76-10655b7382ba\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8"
Oct 30 13:31:43.588705 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:43.588615    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/428b29c0-c277-4e39-ab76-10655b7382ba-istio-envoy\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8\" (UID: \"428b29c0-c277-4e39-ab76-10655b7382ba\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8"
Oct 30 13:31:43.588705 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:43.588697    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmmrl\" (UniqueName: \"kubernetes.io/projected/428b29c0-c277-4e39-ab76-10655b7382ba-kube-api-access-vmmrl\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8\" (UID: \"428b29c0-c277-4e39-ab76-10655b7382ba\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8"
Oct 30 13:31:43.588934 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:43.588743    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/428b29c0-c277-4e39-ab76-10655b7382ba-workload-socket\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8\" (UID: \"428b29c0-c277-4e39-ab76-10655b7382ba\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8"
Oct 30 13:31:43.588934 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:43.588775    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/428b29c0-c277-4e39-ab76-10655b7382ba-istio-token\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8\" (UID: \"428b29c0-c277-4e39-ab76-10655b7382ba\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8"
Oct 30 13:31:43.588934 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:43.588823    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/428b29c0-c277-4e39-ab76-10655b7382ba-workload-certs\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8\" (UID: \"428b29c0-c277-4e39-ab76-10655b7382ba\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8"
Oct 30 13:31:43.588934 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:43.588880    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/428b29c0-c277-4e39-ab76-10655b7382ba-istio-podinfo\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8\" (UID: \"428b29c0-c277-4e39-ab76-10655b7382ba\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8"
Oct 30 13:31:43.588934 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:43.588919    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/428b29c0-c277-4e39-ab76-10655b7382ba-credential-socket\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8\" (UID: \"428b29c0-c277-4e39-ab76-10655b7382ba\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8"
Oct 30 13:31:43.589237 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:43.588972    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/428b29c0-c277-4e39-ab76-10655b7382ba-istio-data\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8\" (UID: \"428b29c0-c277-4e39-ab76-10655b7382ba\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8"
Oct 30 13:31:43.589331 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:43.589304    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/428b29c0-c277-4e39-ab76-10655b7382ba-workload-certs\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8\" (UID: \"428b29c0-c277-4e39-ab76-10655b7382ba\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8"
Oct 30 13:31:43.589402 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:43.589330    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/428b29c0-c277-4e39-ab76-10655b7382ba-credential-socket\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8\" (UID: \"428b29c0-c277-4e39-ab76-10655b7382ba\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8"
Oct 30 13:31:43.589462 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:43.589415    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/428b29c0-c277-4e39-ab76-10655b7382ba-istio-data\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8\" (UID: \"428b29c0-c277-4e39-ab76-10655b7382ba\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8"
Oct 30 13:31:43.589545 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:43.589526    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/428b29c0-c277-4e39-ab76-10655b7382ba-workload-socket\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8\" (UID: \"428b29c0-c277-4e39-ab76-10655b7382ba\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8"
Oct 30 13:31:43.589602 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:43.589550    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/428b29c0-c277-4e39-ab76-10655b7382ba-istiod-ca-cert\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8\" (UID: \"428b29c0-c277-4e39-ab76-10655b7382ba\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8"
Oct 30 13:31:43.590085 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:43.590067    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/428b29c0-c277-4e39-ab76-10655b7382ba-istiod-ca-cert\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8\" (UID: \"428b29c0-c277-4e39-ab76-10655b7382ba\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8"
Oct 30 13:31:43.591107 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:43.591084    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/428b29c0-c277-4e39-ab76-10655b7382ba-istio-envoy\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8\" (UID: \"428b29c0-c277-4e39-ab76-10655b7382ba\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8"
Oct 30 13:31:43.591594 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:43.591565    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/428b29c0-c277-4e39-ab76-10655b7382ba-istio-podinfo\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8\" (UID: \"428b29c0-c277-4e39-ab76-10655b7382ba\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8"
Oct 30 13:31:43.609099 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:43.609074    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/428b29c0-c277-4e39-ab76-10655b7382ba-istio-token\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8\" (UID: \"428b29c0-c277-4e39-ab76-10655b7382ba\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8"
Oct 30 13:31:43.609570 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:43.609552    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmmrl\" (UniqueName: \"kubernetes.io/projected/428b29c0-c277-4e39-ab76-10655b7382ba-kube-api-access-vmmrl\") pod \"e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8\" (UID: \"428b29c0-c277-4e39-ab76-10655b7382ba\") " pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8"
Oct 30 13:31:43.655226 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:43.655190    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8"
Oct 30 13:31:43.982005 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:43.981967    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8"]
Oct 30 13:31:43.982694 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:31:43.982663    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod428b29c0_c277_4e39_ab76_10655b7382ba.slice/crio-8307294254d0433a2e0be356147c0ba9f505943dda9e995dfe016364b27dd7d4 WatchSource:0}: Error finding container 8307294254d0433a2e0be356147c0ba9f505943dda9e995dfe016364b27dd7d4: Status 404 returned error can't find the container with id 8307294254d0433a2e0be356147c0ba9f505943dda9e995dfe016364b27dd7d4
Oct 30 13:31:43.985625 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:43.985583    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:31:43.985718 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:43.985680    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:31:43.985770 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:43.985729    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:31:44.343601 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:44.343499    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8" event={"ID":"428b29c0-c277-4e39-ab76-10655b7382ba","Type":"ContainerStarted","Data":"105319a27172cc8133bb2aaee44b096dc497b2ea3db368c2373fb0939178a819"}
Oct 30 13:31:44.343601 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:44.343574    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8" event={"ID":"428b29c0-c277-4e39-ab76-10655b7382ba","Type":"ContainerStarted","Data":"8307294254d0433a2e0be356147c0ba9f505943dda9e995dfe016364b27dd7d4"}
Oct 30 13:31:44.393756 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:44.393691    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8" podStartSLOduration=1.393676649 podStartE2EDuration="1.393676649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:31:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:31:44.38909737 +0000 UTC m=+1723.536364612" watchObservedRunningTime="2026-10-30 13:31:44.393676649 +0000 UTC m=+1723.540943958"
Oct 30 13:31:44.410967 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:44.410932    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-f45f04/autores-model-f45f04-kserve-854dc88d7c-lt9q8"
Oct 30 13:31:44.430419 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:44.430383    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-f45f04/autores-model-f45f04-kserve-854dc88d7c-lt9q8"
Oct 30 13:31:44.656008 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:44.655905    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8"
Oct 30 13:31:45.381275 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:45.381242    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-f45f04/autores-model-f45f04-kserve-854dc88d7c-lt9q8"
Oct 30 13:31:45.662522 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:45.662428    5093 patch_prober.go:29] interesting pod/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 13:31:45.662522 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:45.662492    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8" podUID="428b29c0-c277-4e39-ab76-10655b7382ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 13:31:46.662976 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:46.662939    5093 patch_prober.go:29] interesting pod/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 13:31:46.663621 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:46.662997    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8" podUID="428b29c0-c277-4e39-ab76-10655b7382ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 13:31:47.656157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:47.655831    5093 patch_prober.go:29] interesting pod/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 13:31:47.656157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:47.655918    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8" podUID="428b29c0-c277-4e39-ab76-10655b7382ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 13:31:48.657590 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:48.657547    5093 patch_prober.go:29] interesting pod/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 13:31:48.658118 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:48.657649    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8" podUID="428b29c0-c277-4e39-ab76-10655b7382ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 13:31:49.656646 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:49.656594    5093 patch_prober.go:29] interesting pod/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 13:31:49.656874 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:49.656671    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8" podUID="428b29c0-c277-4e39-ab76-10655b7382ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 13:31:50.656199 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:50.656147    5093 patch_prober.go:29] interesting pod/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 13:31:50.656733 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:50.656223    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8" podUID="428b29c0-c277-4e39-ab76-10655b7382ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 13:31:51.174975 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:31:51.174928    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:31:51.662290 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:51.662192    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8"
Oct 30 13:31:51.662774 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:51.662714    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8"
Oct 30 13:31:51.663665 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:51.663638    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8"
Oct 30 13:31:58.384548 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:58.384508    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs"]
Oct 30 13:31:58.384981 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:31:58.384725    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs" podUID="c417d75a-c5bf-42fa-8aff-335a63c1295e" containerName="istio-proxy" containerID="cri-o://1b5bad4cf12c2a37eacf7e5769c7faad6086bc4e3db76a8e98573fb316c3d9e9" gracePeriod=30
Oct 30 13:32:03.589607 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:03.589562    5093 generic.go:356] "Generic (PLEG): container finished" podID="c417d75a-c5bf-42fa-8aff-335a63c1295e" containerID="1b5bad4cf12c2a37eacf7e5769c7faad6086bc4e3db76a8e98573fb316c3d9e9" exitCode=0
Oct 30 13:32:03.590114 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:03.589627    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs" event={"ID":"c417d75a-c5bf-42fa-8aff-335a63c1295e","Type":"ContainerDied","Data":"1b5bad4cf12c2a37eacf7e5769c7faad6086bc4e3db76a8e98573fb316c3d9e9"}
Oct 30 13:32:03.590114 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:03.589722    5093 scope.go:122] "RemoveContainer" containerID="ebce1b79b819e2495a9dc6584f37315e5b653389f7a936ae939e2c46153e4daf"
Oct 30 13:32:03.689515 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:03.689485    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs"
Oct 30 13:32:03.886993 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:03.886953    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c417d75a-c5bf-42fa-8aff-335a63c1295e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c417d75a-c5bf-42fa-8aff-335a63c1295e-credential-socket\") pod \"c417d75a-c5bf-42fa-8aff-335a63c1295e\" (UID: \"c417d75a-c5bf-42fa-8aff-335a63c1295e\") "
Oct 30 13:32:03.887161 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:03.887004    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c417d75a-c5bf-42fa-8aff-335a63c1295e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c417d75a-c5bf-42fa-8aff-335a63c1295e-workload-certs\") pod \"c417d75a-c5bf-42fa-8aff-335a63c1295e\" (UID: \"c417d75a-c5bf-42fa-8aff-335a63c1295e\") "
Oct 30 13:32:03.887161 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:03.887039    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c417d75a-c5bf-42fa-8aff-335a63c1295e-kube-api-access-hjf2s\" (UniqueName: \"kubernetes.io/projected/c417d75a-c5bf-42fa-8aff-335a63c1295e-kube-api-access-hjf2s\") pod \"c417d75a-c5bf-42fa-8aff-335a63c1295e\" (UID: \"c417d75a-c5bf-42fa-8aff-335a63c1295e\") "
Oct 30 13:32:03.887264 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:03.887124    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c417d75a-c5bf-42fa-8aff-335a63c1295e-istio-token\" (UniqueName: \"kubernetes.io/projected/c417d75a-c5bf-42fa-8aff-335a63c1295e-istio-token\") pod \"c417d75a-c5bf-42fa-8aff-335a63c1295e\" (UID: \"c417d75a-c5bf-42fa-8aff-335a63c1295e\") "
Oct 30 13:32:03.887308 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:03.887268    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c417d75a-c5bf-42fa-8aff-335a63c1295e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c417d75a-c5bf-42fa-8aff-335a63c1295e-workload-socket\") pod \"c417d75a-c5bf-42fa-8aff-335a63c1295e\" (UID: \"c417d75a-c5bf-42fa-8aff-335a63c1295e\") "
Oct 30 13:32:03.887382 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:03.887351    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c417d75a-c5bf-42fa-8aff-335a63c1295e-credential-socket" pod "c417d75a-c5bf-42fa-8aff-335a63c1295e" (UID: "c417d75a-c5bf-42fa-8aff-335a63c1295e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:32:03.887382 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:03.887374    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c417d75a-c5bf-42fa-8aff-335a63c1295e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c417d75a-c5bf-42fa-8aff-335a63c1295e-istio-envoy\") pod \"c417d75a-c5bf-42fa-8aff-335a63c1295e\" (UID: \"c417d75a-c5bf-42fa-8aff-335a63c1295e\") "
Oct 30 13:32:03.887522 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:03.887363    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c417d75a-c5bf-42fa-8aff-335a63c1295e-workload-certs" pod "c417d75a-c5bf-42fa-8aff-335a63c1295e" (UID: "c417d75a-c5bf-42fa-8aff-335a63c1295e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:32:03.887522 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:03.887421    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c417d75a-c5bf-42fa-8aff-335a63c1295e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c417d75a-c5bf-42fa-8aff-335a63c1295e-istiod-ca-cert\") pod \"c417d75a-c5bf-42fa-8aff-335a63c1295e\" (UID: \"c417d75a-c5bf-42fa-8aff-335a63c1295e\") "
Oct 30 13:32:03.887522 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:03.887450    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c417d75a-c5bf-42fa-8aff-335a63c1295e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c417d75a-c5bf-42fa-8aff-335a63c1295e-istio-podinfo\") pod \"c417d75a-c5bf-42fa-8aff-335a63c1295e\" (UID: \"c417d75a-c5bf-42fa-8aff-335a63c1295e\") "
Oct 30 13:32:03.887684 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:03.887538    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c417d75a-c5bf-42fa-8aff-335a63c1295e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c417d75a-c5bf-42fa-8aff-335a63c1295e-istio-data\") pod \"c417d75a-c5bf-42fa-8aff-335a63c1295e\" (UID: \"c417d75a-c5bf-42fa-8aff-335a63c1295e\") "
Oct 30 13:32:03.887684 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:03.887567    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c417d75a-c5bf-42fa-8aff-335a63c1295e-workload-socket" pod "c417d75a-c5bf-42fa-8aff-335a63c1295e" (UID: "c417d75a-c5bf-42fa-8aff-335a63c1295e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:32:03.887816 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:03.887780    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c417d75a-c5bf-42fa-8aff-335a63c1295e-istiod-ca-cert" pod "c417d75a-c5bf-42fa-8aff-335a63c1295e" (UID: "c417d75a-c5bf-42fa-8aff-335a63c1295e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:32:03.887875 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:03.887843    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c417d75a-c5bf-42fa-8aff-335a63c1295e-istio-data" pod "c417d75a-c5bf-42fa-8aff-335a63c1295e" (UID: "c417d75a-c5bf-42fa-8aff-335a63c1295e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:32:03.888399 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:03.888372    5093 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c417d75a-c5bf-42fa-8aff-335a63c1295e-workload-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:32:03.888524 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:03.888402    5093 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c417d75a-c5bf-42fa-8aff-335a63c1295e-istiod-ca-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:32:03.888524 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:03.888418    5093 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c417d75a-c5bf-42fa-8aff-335a63c1295e-istio-data\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:32:03.888524 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:03.888432    5093 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c417d75a-c5bf-42fa-8aff-335a63c1295e-credential-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:32:03.888524 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:03.888448    5093 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c417d75a-c5bf-42fa-8aff-335a63c1295e-workload-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:32:03.889696 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:03.889670    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c417d75a-c5bf-42fa-8aff-335a63c1295e-istio-podinfo" pod "c417d75a-c5bf-42fa-8aff-335a63c1295e" (UID: "c417d75a-c5bf-42fa-8aff-335a63c1295e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:32:03.889812 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:03.889729    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c417d75a-c5bf-42fa-8aff-335a63c1295e-istio-token" pod "c417d75a-c5bf-42fa-8aff-335a63c1295e" (UID: "c417d75a-c5bf-42fa-8aff-335a63c1295e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:32:03.889812 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:03.889769    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c417d75a-c5bf-42fa-8aff-335a63c1295e-kube-api-access-hjf2s" pod "c417d75a-c5bf-42fa-8aff-335a63c1295e" (UID: "c417d75a-c5bf-42fa-8aff-335a63c1295e"). InnerVolumeSpecName "kube-api-access-hjf2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:32:03.889812 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:03.889794    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c417d75a-c5bf-42fa-8aff-335a63c1295e-istio-envoy" pod "c417d75a-c5bf-42fa-8aff-335a63c1295e" (UID: "c417d75a-c5bf-42fa-8aff-335a63c1295e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:32:03.989937 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:03.989900    5093 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c417d75a-c5bf-42fa-8aff-335a63c1295e-istio-envoy\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:32:03.989937 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:03.989931    5093 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c417d75a-c5bf-42fa-8aff-335a63c1295e-istio-podinfo\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:32:03.989937 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:03.989941    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjf2s\" (UniqueName: \"kubernetes.io/projected/c417d75a-c5bf-42fa-8aff-335a63c1295e-kube-api-access-hjf2s\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:32:03.990198 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:03.989951    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c417d75a-c5bf-42fa-8aff-335a63c1295e-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:32:04.601320 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:04.601291    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs"
Oct 30 13:32:04.601931 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:04.601327    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs" event={"ID":"c417d75a-c5bf-42fa-8aff-335a63c1295e","Type":"ContainerDied","Data":"99226e58d1d627da6e0464fcad298ea1921e083fa49f00e38ef30aa0b55d6d19"}
Oct 30 13:32:04.602092 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:04.602078    5093 scope.go:122] "RemoveContainer" containerID="1b5bad4cf12c2a37eacf7e5769c7faad6086bc4e3db76a8e98573fb316c3d9e9"
Oct 30 13:32:04.676058 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:04.676015    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs"]
Oct 30 13:32:04.688468 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:04.688431    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-knngs"]
Oct 30 13:32:05.001082 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:05.001044    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c417d75a-c5bf-42fa-8aff-335a63c1295e" path="/var/lib/kubelet/pods/c417d75a-c5bf-42fa-8aff-335a63c1295e/volumes"
Oct 30 13:32:09.751154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:09.750144    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-f12b80/auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg"]
Oct 30 13:32:09.751704 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:09.751381    5093 cpu_manager.go:414] "containerMap: removing container" podUID="c417d75a-c5bf-42fa-8aff-335a63c1295e" containerName="istio-proxy"
Oct 30 13:32:09.751704 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:09.751406    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c417d75a-c5bf-42fa-8aff-335a63c1295e" containerName="istio-proxy"
Oct 30 13:32:09.751704 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:09.751437    5093 cpu_manager.go:414] "containerMap: removing container" podUID="c417d75a-c5bf-42fa-8aff-335a63c1295e" containerName="istio-proxy"
Oct 30 13:32:09.751704 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:09.751445    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c417d75a-c5bf-42fa-8aff-335a63c1295e" containerName="istio-proxy"
Oct 30 13:32:09.751704 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:09.751650    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="c417d75a-c5bf-42fa-8aff-335a63c1295e" containerName="istio-proxy"
Oct 30 13:32:09.751704 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:09.751675    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="c417d75a-c5bf-42fa-8aff-335a63c1295e" containerName="istio-proxy"
Oct 30 13:32:09.773154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:09.758595    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-f12b80/auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg"
Oct 30 13:32:09.782933 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:09.782880    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-f12b80/auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg"]
Oct 30 13:32:09.784297 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:09.784257    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vr8m\" (UniqueName: \"kubernetes.io/projected/98a0175c-d89f-44c3-aae9-4585f86974f8-kube-api-access-8vr8m\") pod \"auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg\" (UID: \"98a0175c-d89f-44c3-aae9-4585f86974f8\") " pod="ai-tenant-e2e-shared-b-w4-f12b80/auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg"
Oct 30 13:32:09.784740 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:09.784715    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98a0175c-d89f-44c3-aae9-4585f86974f8-tls-certs\") pod \"auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg\" (UID: \"98a0175c-d89f-44c3-aae9-4585f86974f8\") " pod="ai-tenant-e2e-shared-b-w4-f12b80/auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg"
Oct 30 13:32:09.785057 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:09.785037    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98a0175c-d89f-44c3-aae9-4585f86974f8-tmp-dir\") pod \"auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg\" (UID: \"98a0175c-d89f-44c3-aae9-4585f86974f8\") " pod="ai-tenant-e2e-shared-b-w4-f12b80/auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg"
Oct 30 13:32:09.785261 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:09.785242    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98a0175c-d89f-44c3-aae9-4585f86974f8-home\") pod \"auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg\" (UID: \"98a0175c-d89f-44c3-aae9-4585f86974f8\") " pod="ai-tenant-e2e-shared-b-w4-f12b80/auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg"
Oct 30 13:32:09.785365 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:09.785287    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98a0175c-d89f-44c3-aae9-4585f86974f8-model-cache\") pod \"auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg\" (UID: \"98a0175c-d89f-44c3-aae9-4585f86974f8\") " pod="ai-tenant-e2e-shared-b-w4-f12b80/auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg"
Oct 30 13:32:09.785428 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:09.785379    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98a0175c-d89f-44c3-aae9-4585f86974f8-dshm\") pod \"auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg\" (UID: \"98a0175c-d89f-44c3-aae9-4585f86974f8\") " pod="ai-tenant-e2e-shared-b-w4-f12b80/auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg"
Oct 30 13:32:09.786656 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:09.786637    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-f12b80\"/\"kube-root-ca.crt\""
Oct 30 13:32:09.815509 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:09.814485    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-f12b80\"/\"auth-test-model-f12b80-kserve-self-signed-certs\""
Oct 30 13:32:09.815509 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:09.814557    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-f12b80\"/\"default-dockercfg-wglcq\""
Oct 30 13:32:09.815509 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:09.814797    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-f12b80\"/\"openshift-service-ca.crt\""
Oct 30 13:32:09.886823 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:09.886777    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98a0175c-d89f-44c3-aae9-4585f86974f8-tls-certs\") pod \"auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg\" (UID: \"98a0175c-d89f-44c3-aae9-4585f86974f8\") " pod="ai-tenant-e2e-shared-b-w4-f12b80/auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg"
Oct 30 13:32:09.887010 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:09.886909    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98a0175c-d89f-44c3-aae9-4585f86974f8-tmp-dir\") pod \"auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg\" (UID: \"98a0175c-d89f-44c3-aae9-4585f86974f8\") " pod="ai-tenant-e2e-shared-b-w4-f12b80/auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg"
Oct 30 13:32:09.887010 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:09.886945    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98a0175c-d89f-44c3-aae9-4585f86974f8-home\") pod \"auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg\" (UID: \"98a0175c-d89f-44c3-aae9-4585f86974f8\") " pod="ai-tenant-e2e-shared-b-w4-f12b80/auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg"
Oct 30 13:32:09.887010 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:09.886983    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98a0175c-d89f-44c3-aae9-4585f86974f8-model-cache\") pod \"auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg\" (UID: \"98a0175c-d89f-44c3-aae9-4585f86974f8\") " pod="ai-tenant-e2e-shared-b-w4-f12b80/auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg"
Oct 30 13:32:09.887267 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:09.887060    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98a0175c-d89f-44c3-aae9-4585f86974f8-dshm\") pod \"auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg\" (UID: \"98a0175c-d89f-44c3-aae9-4585f86974f8\") " pod="ai-tenant-e2e-shared-b-w4-f12b80/auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg"
Oct 30 13:32:09.887267 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:09.887147    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vr8m\" (UniqueName: \"kubernetes.io/projected/98a0175c-d89f-44c3-aae9-4585f86974f8-kube-api-access-8vr8m\") pod \"auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg\" (UID: \"98a0175c-d89f-44c3-aae9-4585f86974f8\") " pod="ai-tenant-e2e-shared-b-w4-f12b80/auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg"
Oct 30 13:32:09.887407 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:09.887380    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98a0175c-d89f-44c3-aae9-4585f86974f8-tmp-dir\") pod \"auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg\" (UID: \"98a0175c-d89f-44c3-aae9-4585f86974f8\") " pod="ai-tenant-e2e-shared-b-w4-f12b80/auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg"
Oct 30 13:32:09.887534 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:09.887442    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98a0175c-d89f-44c3-aae9-4585f86974f8-home\") pod \"auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg\" (UID: \"98a0175c-d89f-44c3-aae9-4585f86974f8\") " pod="ai-tenant-e2e-shared-b-w4-f12b80/auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg"
Oct 30 13:32:09.887721 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:09.887702    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98a0175c-d89f-44c3-aae9-4585f86974f8-model-cache\") pod \"auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg\" (UID: \"98a0175c-d89f-44c3-aae9-4585f86974f8\") " pod="ai-tenant-e2e-shared-b-w4-f12b80/auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg"
Oct 30 13:32:09.889541 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:09.889519    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98a0175c-d89f-44c3-aae9-4585f86974f8-tls-certs\") pod \"auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg\" (UID: \"98a0175c-d89f-44c3-aae9-4585f86974f8\") " pod="ai-tenant-e2e-shared-b-w4-f12b80/auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg"
Oct 30 13:32:09.889664 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:09.889643    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98a0175c-d89f-44c3-aae9-4585f86974f8-dshm\") pod \"auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg\" (UID: \"98a0175c-d89f-44c3-aae9-4585f86974f8\") " pod="ai-tenant-e2e-shared-b-w4-f12b80/auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg"
Oct 30 13:32:09.907626 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:09.907586    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vr8m\" (UniqueName: \"kubernetes.io/projected/98a0175c-d89f-44c3-aae9-4585f86974f8-kube-api-access-8vr8m\") pod \"auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg\" (UID: \"98a0175c-d89f-44c3-aae9-4585f86974f8\") " pod="ai-tenant-e2e-shared-b-w4-f12b80/auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg"
Oct 30 13:32:10.122648 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:10.122552    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-f12b80/auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg"
Oct 30 13:32:10.519509 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:10.519476    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-f12b80/auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg"]
Oct 30 13:32:10.521577 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:32:10.521549    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98a0175c_d89f_44c3_aae9_4585f86974f8.slice/crio-0eb65abd551ef6f7222ca322fd4ceb5a5e337911eab528a9b8edd274c575a0c6 WatchSource:0}: Error finding container 0eb65abd551ef6f7222ca322fd4ceb5a5e337911eab528a9b8edd274c575a0c6: Status 404 returned error can't find the container with id 0eb65abd551ef6f7222ca322fd4ceb5a5e337911eab528a9b8edd274c575a0c6
Oct 30 13:32:10.689329 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:10.689098    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-f12b80/auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg" event={"ID":"98a0175c-d89f-44c3-aae9-4585f86974f8","Type":"ContainerStarted","Data":"157abe299b737087a241aa5a703bc2e47cb63315377aafd05a2a3c512dec4882"}
Oct 30 13:32:10.689329 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:10.689188    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-f12b80/auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg" event={"ID":"98a0175c-d89f-44c3-aae9-4585f86974f8","Type":"ContainerStarted","Data":"0eb65abd551ef6f7222ca322fd4ceb5a5e337911eab528a9b8edd274c575a0c6"}
Oct 30 13:32:10.734086 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:10.734029    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-f12b80/auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg" podStartSLOduration=1.73400987 podStartE2EDuration="1.73400987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:32:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:32:10.73241061 +0000 UTC m=+1749.879677851" watchObservedRunningTime="2026-10-30 13:32:10.73400987 +0000 UTC m=+1749.881277110"
Oct 30 13:32:11.123588 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:11.123555    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-f12b80/auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg"
Oct 30 13:32:15.073183 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:15.067865    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-f45f04/autores-model-f45f04-kserve-854dc88d7c-lt9q8"]
Oct 30 13:32:20.123624 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:20.123586    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-f12b80/auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg"
Oct 30 13:32:20.136780 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:20.136749    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-f12b80/auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg"
Oct 30 13:32:20.796422 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:20.796391    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-f12b80/auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg"
Oct 30 13:32:21.170717 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:32:21.170625    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:32:30.095866 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:30.095816    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-f45f04/autores-model-f45f04-kserve-854dc88d7c-lt9q8" podUID="a5f4950f-ab65-4763-9d96-e4210dd6e955" containerName="main" containerID="cri-o://2c90a419c4306dc73ced146ea2d4297eff824844f56d2ad4c46747dce16620f9" gracePeriod=45
Oct 30 13:32:30.417579 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:30.417550    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-f45f04/autores-model-f45f04-kserve-854dc88d7c-lt9q8"
Oct 30 13:32:30.566562 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:30.566511    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f4950f-ab65-4763-9d96-e4210dd6e955-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5f4950f-ab65-4763-9d96-e4210dd6e955-dshm\") pod \"a5f4950f-ab65-4763-9d96-e4210dd6e955\" (UID: \"a5f4950f-ab65-4763-9d96-e4210dd6e955\") "
Oct 30 13:32:30.566763 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:30.566619    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5f4950f-ab65-4763-9d96-e4210dd6e955-kube-api-access-787rc\" (UniqueName: \"kubernetes.io/projected/a5f4950f-ab65-4763-9d96-e4210dd6e955-kube-api-access-787rc\") pod \"a5f4950f-ab65-4763-9d96-e4210dd6e955\" (UID: \"a5f4950f-ab65-4763-9d96-e4210dd6e955\") "
Oct 30 13:32:30.566763 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:30.566682    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f4950f-ab65-4763-9d96-e4210dd6e955-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5f4950f-ab65-4763-9d96-e4210dd6e955-tmp-dir\") pod \"a5f4950f-ab65-4763-9d96-e4210dd6e955\" (UID: \"a5f4950f-ab65-4763-9d96-e4210dd6e955\") "
Oct 30 13:32:30.566763 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:30.566717    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f4950f-ab65-4763-9d96-e4210dd6e955-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5f4950f-ab65-4763-9d96-e4210dd6e955-model-cache\") pod \"a5f4950f-ab65-4763-9d96-e4210dd6e955\" (UID: \"a5f4950f-ab65-4763-9d96-e4210dd6e955\") "
Oct 30 13:32:30.566763 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:30.566757    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5f4950f-ab65-4763-9d96-e4210dd6e955-tls-certs\" (UniqueName: \"kubernetes.io/secret/a5f4950f-ab65-4763-9d96-e4210dd6e955-tls-certs\") pod \"a5f4950f-ab65-4763-9d96-e4210dd6e955\" (UID: \"a5f4950f-ab65-4763-9d96-e4210dd6e955\") "
Oct 30 13:32:30.567001 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:30.566909    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f4950f-ab65-4763-9d96-e4210dd6e955-home\" (UniqueName: \"kubernetes.io/empty-dir/a5f4950f-ab65-4763-9d96-e4210dd6e955-home\") pod \"a5f4950f-ab65-4763-9d96-e4210dd6e955\" (UID: \"a5f4950f-ab65-4763-9d96-e4210dd6e955\") "
Oct 30 13:32:30.567056 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:30.567009    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f4950f-ab65-4763-9d96-e4210dd6e955-model-cache" pod "a5f4950f-ab65-4763-9d96-e4210dd6e955" (UID: "a5f4950f-ab65-4763-9d96-e4210dd6e955"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:32:30.567379 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:30.567348    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f4950f-ab65-4763-9d96-e4210dd6e955-tmp-dir" pod "a5f4950f-ab65-4763-9d96-e4210dd6e955" (UID: "a5f4950f-ab65-4763-9d96-e4210dd6e955"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:32:30.567595 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:30.567360    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f4950f-ab65-4763-9d96-e4210dd6e955-home" pod "a5f4950f-ab65-4763-9d96-e4210dd6e955" (UID: "a5f4950f-ab65-4763-9d96-e4210dd6e955"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:32:30.567750 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:30.567727    5093 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5f4950f-ab65-4763-9d96-e4210dd6e955-tmp-dir\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:32:30.567831 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:30.567755    5093 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5f4950f-ab65-4763-9d96-e4210dd6e955-model-cache\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:32:30.567831 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:30.567770    5093 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5f4950f-ab65-4763-9d96-e4210dd6e955-home\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:32:30.569516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:30.569489    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f4950f-ab65-4763-9d96-e4210dd6e955-kube-api-access-787rc" pod "a5f4950f-ab65-4763-9d96-e4210dd6e955" (UID: "a5f4950f-ab65-4763-9d96-e4210dd6e955"). InnerVolumeSpecName "kube-api-access-787rc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:32:30.569667 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:30.569649    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f4950f-ab65-4763-9d96-e4210dd6e955-dshm" pod "a5f4950f-ab65-4763-9d96-e4210dd6e955" (UID: "a5f4950f-ab65-4763-9d96-e4210dd6e955"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:32:30.569795 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:30.569771    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5f4950f-ab65-4763-9d96-e4210dd6e955-tls-certs" pod "a5f4950f-ab65-4763-9d96-e4210dd6e955" (UID: "a5f4950f-ab65-4763-9d96-e4210dd6e955"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:32:30.669423 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:30.669330    5093 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5f4950f-ab65-4763-9d96-e4210dd6e955-dshm\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:32:30.669423 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:30.669358    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-787rc\" (UniqueName: \"kubernetes.io/projected/a5f4950f-ab65-4763-9d96-e4210dd6e955-kube-api-access-787rc\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:32:30.669423 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:30.669370    5093 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5f4950f-ab65-4763-9d96-e4210dd6e955-tls-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:32:30.892989 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:30.892955    5093 generic.go:356] "Generic (PLEG): container finished" podID="a5f4950f-ab65-4763-9d96-e4210dd6e955" containerID="2c90a419c4306dc73ced146ea2d4297eff824844f56d2ad4c46747dce16620f9" exitCode=0
Oct 30 13:32:30.893182 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:30.893015    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-f45f04/autores-model-f45f04-kserve-854dc88d7c-lt9q8"
Oct 30 13:32:30.893182 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:30.893041    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-f45f04/autores-model-f45f04-kserve-854dc88d7c-lt9q8" event={"ID":"a5f4950f-ab65-4763-9d96-e4210dd6e955","Type":"ContainerDied","Data":"2c90a419c4306dc73ced146ea2d4297eff824844f56d2ad4c46747dce16620f9"}
Oct 30 13:32:30.893182 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:30.893091    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-f45f04/autores-model-f45f04-kserve-854dc88d7c-lt9q8" event={"ID":"a5f4950f-ab65-4763-9d96-e4210dd6e955","Type":"ContainerDied","Data":"56c7526c5fd614d46b8897c26b035d67b6621db2391c3d723a91ca22a744abef"}
Oct 30 13:32:30.893182 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:30.893104    5093 scope.go:122] "RemoveContainer" containerID="2c90a419c4306dc73ced146ea2d4297eff824844f56d2ad4c46747dce16620f9"
Oct 30 13:32:30.908946 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:30.908923    5093 scope.go:122] "RemoveContainer" containerID="2c90a419c4306dc73ced146ea2d4297eff824844f56d2ad4c46747dce16620f9"
Oct 30 13:32:30.909258 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:32:30.909236    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c90a419c4306dc73ced146ea2d4297eff824844f56d2ad4c46747dce16620f9\": container with ID starting with 2c90a419c4306dc73ced146ea2d4297eff824844f56d2ad4c46747dce16620f9 not found: ID does not exist" containerID="2c90a419c4306dc73ced146ea2d4297eff824844f56d2ad4c46747dce16620f9"
Oct 30 13:32:30.909305 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:30.909270    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c90a419c4306dc73ced146ea2d4297eff824844f56d2ad4c46747dce16620f9"} err="failed to get container status \"2c90a419c4306dc73ced146ea2d4297eff824844f56d2ad4c46747dce16620f9\": rpc error: code = NotFound desc = could not find container \"2c90a419c4306dc73ced146ea2d4297eff824844f56d2ad4c46747dce16620f9\": container with ID starting with 2c90a419c4306dc73ced146ea2d4297eff824844f56d2ad4c46747dce16620f9 not found: ID does not exist"
Oct 30 13:32:30.951156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:30.951021    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-f45f04/autores-model-f45f04-kserve-854dc88d7c-lt9q8"]
Oct 30 13:32:30.957496 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:30.957447    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-f45f04/autores-model-f45f04-kserve-854dc88d7c-lt9q8"]
Oct 30 13:32:31.001591 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:31.001555    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5f4950f-ab65-4763-9d96-e4210dd6e955" path="/var/lib/kubelet/pods/a5f4950f-ab65-4763-9d96-e4210dd6e955/volumes"
Oct 30 13:32:31.365312 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:31.365210    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-autores-f45f04/autores-model-f45f04-kserve-854dc88d7c-lt9q8" podUID="a5f4950f-ab65-4763-9d96-e4210dd6e955" containerName="main" probeResult="failure" output="Get \"https://10.132.0.138:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:32:39.777074 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:39.777032    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-f7ee8a/auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz"]
Oct 30 13:32:39.777837 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:39.777818    5093 cpu_manager.go:414] "containerMap: removing container" podUID="a5f4950f-ab65-4763-9d96-e4210dd6e955" containerName="main"
Oct 30 13:32:39.777884 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:39.777839    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5f4950f-ab65-4763-9d96-e4210dd6e955" containerName="main"
Oct 30 13:32:39.777968 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:39.777959    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5f4950f-ab65-4763-9d96-e4210dd6e955" containerName="main"
Oct 30 13:32:39.783674 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:39.783647    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-f7ee8a/auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz"
Oct 30 13:32:39.806559 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:39.806520    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt5tf\" (UniqueName: \"kubernetes.io/projected/ed392ef1-7739-4131-9e4b-9568d9e68801-kube-api-access-bt5tf\") pod \"auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz\" (UID: \"ed392ef1-7739-4131-9e4b-9568d9e68801\") " pod="ai-tenant-e2e-shared-a-w1-f7ee8a/auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz"
Oct 30 13:32:39.806737 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:39.806576    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed392ef1-7739-4131-9e4b-9568d9e68801-model-cache\") pod \"auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz\" (UID: \"ed392ef1-7739-4131-9e4b-9568d9e68801\") " pod="ai-tenant-e2e-shared-a-w1-f7ee8a/auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz"
Oct 30 13:32:39.806737 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:39.806641    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed392ef1-7739-4131-9e4b-9568d9e68801-tls-certs\") pod \"auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz\" (UID: \"ed392ef1-7739-4131-9e4b-9568d9e68801\") " pod="ai-tenant-e2e-shared-a-w1-f7ee8a/auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz"
Oct 30 13:32:39.806737 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:39.806721    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed392ef1-7739-4131-9e4b-9568d9e68801-dshm\") pod \"auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz\" (UID: \"ed392ef1-7739-4131-9e4b-9568d9e68801\") " pod="ai-tenant-e2e-shared-a-w1-f7ee8a/auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz"
Oct 30 13:32:39.806930 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:39.806780    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed392ef1-7739-4131-9e4b-9568d9e68801-tmp-dir\") pod \"auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz\" (UID: \"ed392ef1-7739-4131-9e4b-9568d9e68801\") " pod="ai-tenant-e2e-shared-a-w1-f7ee8a/auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz"
Oct 30 13:32:39.806930 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:39.806850    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed392ef1-7739-4131-9e4b-9568d9e68801-home\") pod \"auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz\" (UID: \"ed392ef1-7739-4131-9e4b-9568d9e68801\") " pod="ai-tenant-e2e-shared-a-w1-f7ee8a/auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz"
Oct 30 13:32:39.812615 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:39.812585    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-f7ee8a/auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz"]
Oct 30 13:32:39.814987 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:39.814898    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-f7ee8a\"/\"kube-root-ca.crt\""
Oct 30 13:32:39.814987 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:39.814963    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-f7ee8a\"/\"openshift-service-ca.crt\""
Oct 30 13:32:39.814987 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:39.814930    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-f7ee8a\"/\"auth-test-model-f7ee8a-kserve-self-signed-certs\""
Oct 30 13:32:39.815256 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:39.814901    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-f7ee8a\"/\"default-dockercfg-v895p\""
Oct 30 13:32:39.907743 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:39.907698    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed392ef1-7739-4131-9e4b-9568d9e68801-dshm\") pod \"auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz\" (UID: \"ed392ef1-7739-4131-9e4b-9568d9e68801\") " pod="ai-tenant-e2e-shared-a-w1-f7ee8a/auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz"
Oct 30 13:32:39.907933 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:39.907794    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed392ef1-7739-4131-9e4b-9568d9e68801-tmp-dir\") pod \"auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz\" (UID: \"ed392ef1-7739-4131-9e4b-9568d9e68801\") " pod="ai-tenant-e2e-shared-a-w1-f7ee8a/auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz"
Oct 30 13:32:39.907933 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:39.907882    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed392ef1-7739-4131-9e4b-9568d9e68801-home\") pod \"auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz\" (UID: \"ed392ef1-7739-4131-9e4b-9568d9e68801\") " pod="ai-tenant-e2e-shared-a-w1-f7ee8a/auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz"
Oct 30 13:32:39.908031 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:39.908015    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt5tf\" (UniqueName: \"kubernetes.io/projected/ed392ef1-7739-4131-9e4b-9568d9e68801-kube-api-access-bt5tf\") pod \"auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz\" (UID: \"ed392ef1-7739-4131-9e4b-9568d9e68801\") " pod="ai-tenant-e2e-shared-a-w1-f7ee8a/auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz"
Oct 30 13:32:39.908087 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:39.908047    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed392ef1-7739-4131-9e4b-9568d9e68801-model-cache\") pod \"auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz\" (UID: \"ed392ef1-7739-4131-9e4b-9568d9e68801\") " pod="ai-tenant-e2e-shared-a-w1-f7ee8a/auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz"
Oct 30 13:32:39.908229 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:39.908203    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed392ef1-7739-4131-9e4b-9568d9e68801-tls-certs\") pod \"auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz\" (UID: \"ed392ef1-7739-4131-9e4b-9568d9e68801\") " pod="ai-tenant-e2e-shared-a-w1-f7ee8a/auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz"
Oct 30 13:32:39.908509 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:39.908276    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed392ef1-7739-4131-9e4b-9568d9e68801-tmp-dir\") pod \"auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz\" (UID: \"ed392ef1-7739-4131-9e4b-9568d9e68801\") " pod="ai-tenant-e2e-shared-a-w1-f7ee8a/auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz"
Oct 30 13:32:39.908599 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:39.908349    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed392ef1-7739-4131-9e4b-9568d9e68801-home\") pod \"auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz\" (UID: \"ed392ef1-7739-4131-9e4b-9568d9e68801\") " pod="ai-tenant-e2e-shared-a-w1-f7ee8a/auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz"
Oct 30 13:32:39.908599 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:39.908528    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed392ef1-7739-4131-9e4b-9568d9e68801-model-cache\") pod \"auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz\" (UID: \"ed392ef1-7739-4131-9e4b-9568d9e68801\") " pod="ai-tenant-e2e-shared-a-w1-f7ee8a/auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz"
Oct 30 13:32:39.910395 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:39.910370    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed392ef1-7739-4131-9e4b-9568d9e68801-dshm\") pod \"auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz\" (UID: \"ed392ef1-7739-4131-9e4b-9568d9e68801\") " pod="ai-tenant-e2e-shared-a-w1-f7ee8a/auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz"
Oct 30 13:32:39.910841 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:39.910817    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed392ef1-7739-4131-9e4b-9568d9e68801-tls-certs\") pod \"auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz\" (UID: \"ed392ef1-7739-4131-9e4b-9568d9e68801\") " pod="ai-tenant-e2e-shared-a-w1-f7ee8a/auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz"
Oct 30 13:32:39.928539 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:39.928502    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt5tf\" (UniqueName: \"kubernetes.io/projected/ed392ef1-7739-4131-9e4b-9568d9e68801-kube-api-access-bt5tf\") pod \"auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz\" (UID: \"ed392ef1-7739-4131-9e4b-9568d9e68801\") " pod="ai-tenant-e2e-shared-a-w1-f7ee8a/auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz"
Oct 30 13:32:40.111430 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:40.111329    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-f7ee8a/auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz"
Oct 30 13:32:40.443321 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:40.443289    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-f7ee8a/auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz"]
Oct 30 13:32:40.443474 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:32:40.443417    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded392ef1_7739_4131_9e4b_9568d9e68801.slice/crio-56c35a372e7422e742f6e20270e6a93264b503493df03d9f2b1daea1b5cb6de5 WatchSource:0}: Error finding container 56c35a372e7422e742f6e20270e6a93264b503493df03d9f2b1daea1b5cb6de5: Status 404 returned error can't find the container with id 56c35a372e7422e742f6e20270e6a93264b503493df03d9f2b1daea1b5cb6de5
Oct 30 13:32:41.000318 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:41.000280    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-f7ee8a/auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz" event={"ID":"ed392ef1-7739-4131-9e4b-9568d9e68801","Type":"ContainerStarted","Data":"50e24476000696345ec9607a66e83c774dfae3403d41ba7f6d38d0cb96eecbfa"}
Oct 30 13:32:41.000722 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:41.000327    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-f7ee8a/auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz" event={"ID":"ed392ef1-7739-4131-9e4b-9568d9e68801","Type":"ContainerStarted","Data":"56c35a372e7422e742f6e20270e6a93264b503493df03d9f2b1daea1b5cb6de5"}
Oct 30 13:32:41.040850 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:41.040792    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-f7ee8a/auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz" podStartSLOduration=2.040777502 podStartE2EDuration="2.040777502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:32:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:32:41.036595738 +0000 UTC m=+1780.183862979" watchObservedRunningTime="2026-10-30 13:32:41.040777502 +0000 UTC m=+1780.188044742"
Oct 30 13:32:41.112219 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:41.112177    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-f7ee8a/auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz"
Oct 30 13:32:50.111502 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:50.111464    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-f7ee8a/auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz"
Oct 30 13:32:50.124775 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:50.124742    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-f7ee8a/auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz"
Oct 30 13:32:51.102867 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:51.102835    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-f7ee8a/auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz"
Oct 30 13:32:51.191360 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:32:51.191315    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:32:59.383408 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:59.383361    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-29a3f3/auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc"]
Oct 30 13:32:59.396298 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:59.396259    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-29a3f3/auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc"
Oct 30 13:32:59.413230 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:59.410712    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-29a3f3/auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc"]
Oct 30 13:32:59.415786 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:59.415754    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-29a3f3\"/\"kube-root-ca.crt\""
Oct 30 13:32:59.417174 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:59.417105    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-29a3f3\"/\"openshift-service-ca.crt\""
Oct 30 13:32:59.423019 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:59.422986    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlbtw\" (UniqueName: \"kubernetes.io/projected/a8f4bffb-6519-4c67-bb97-115403fafa6c-kube-api-access-qlbtw\") pod \"auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc\" (UID: \"a8f4bffb-6519-4c67-bb97-115403fafa6c\") " pod="ai-tenant-e2e-shared-b-w1-29a3f3/auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc"
Oct 30 13:32:59.423272 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:59.423252    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8f4bffb-6519-4c67-bb97-115403fafa6c-tmp-dir\") pod \"auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc\" (UID: \"a8f4bffb-6519-4c67-bb97-115403fafa6c\") " pod="ai-tenant-e2e-shared-b-w1-29a3f3/auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc"
Oct 30 13:32:59.423362 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:59.423294    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8f4bffb-6519-4c67-bb97-115403fafa6c-dshm\") pod \"auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc\" (UID: \"a8f4bffb-6519-4c67-bb97-115403fafa6c\") " pod="ai-tenant-e2e-shared-b-w1-29a3f3/auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc"
Oct 30 13:32:59.423426 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:59.423406    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8f4bffb-6519-4c67-bb97-115403fafa6c-tls-certs\") pod \"auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc\" (UID: \"a8f4bffb-6519-4c67-bb97-115403fafa6c\") " pod="ai-tenant-e2e-shared-b-w1-29a3f3/auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc"
Oct 30 13:32:59.423479 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:59.423461    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8f4bffb-6519-4c67-bb97-115403fafa6c-model-cache\") pod \"auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc\" (UID: \"a8f4bffb-6519-4c67-bb97-115403fafa6c\") " pod="ai-tenant-e2e-shared-b-w1-29a3f3/auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc"
Oct 30 13:32:59.423591 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:59.423563    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8f4bffb-6519-4c67-bb97-115403fafa6c-home\") pod \"auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc\" (UID: \"a8f4bffb-6519-4c67-bb97-115403fafa6c\") " pod="ai-tenant-e2e-shared-b-w1-29a3f3/auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc"
Oct 30 13:32:59.484517 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:59.484481    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-29a3f3\"/\"auth-test-model-29a3f3-kserve-self-signed-certs\""
Oct 30 13:32:59.484517 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:59.484481    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-29a3f3\"/\"default-dockercfg-7wx22\""
Oct 30 13:32:59.525251 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:59.525212    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlbtw\" (UniqueName: \"kubernetes.io/projected/a8f4bffb-6519-4c67-bb97-115403fafa6c-kube-api-access-qlbtw\") pod \"auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc\" (UID: \"a8f4bffb-6519-4c67-bb97-115403fafa6c\") " pod="ai-tenant-e2e-shared-b-w1-29a3f3/auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc"
Oct 30 13:32:59.525461 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:59.525428    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8f4bffb-6519-4c67-bb97-115403fafa6c-tmp-dir\") pod \"auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc\" (UID: \"a8f4bffb-6519-4c67-bb97-115403fafa6c\") " pod="ai-tenant-e2e-shared-b-w1-29a3f3/auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc"
Oct 30 13:32:59.525535 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:59.525464    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8f4bffb-6519-4c67-bb97-115403fafa6c-dshm\") pod \"auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc\" (UID: \"a8f4bffb-6519-4c67-bb97-115403fafa6c\") " pod="ai-tenant-e2e-shared-b-w1-29a3f3/auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc"
Oct 30 13:32:59.525535 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:59.525527    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8f4bffb-6519-4c67-bb97-115403fafa6c-tls-certs\") pod \"auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc\" (UID: \"a8f4bffb-6519-4c67-bb97-115403fafa6c\") " pod="ai-tenant-e2e-shared-b-w1-29a3f3/auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc"
Oct 30 13:32:59.525650 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:59.525567    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8f4bffb-6519-4c67-bb97-115403fafa6c-model-cache\") pod \"auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc\" (UID: \"a8f4bffb-6519-4c67-bb97-115403fafa6c\") " pod="ai-tenant-e2e-shared-b-w1-29a3f3/auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc"
Oct 30 13:32:59.525650 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:59.525616    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8f4bffb-6519-4c67-bb97-115403fafa6c-home\") pod \"auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc\" (UID: \"a8f4bffb-6519-4c67-bb97-115403fafa6c\") " pod="ai-tenant-e2e-shared-b-w1-29a3f3/auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc"
Oct 30 13:32:59.525960 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:59.525921    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8f4bffb-6519-4c67-bb97-115403fafa6c-tmp-dir\") pod \"auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc\" (UID: \"a8f4bffb-6519-4c67-bb97-115403fafa6c\") " pod="ai-tenant-e2e-shared-b-w1-29a3f3/auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc"
Oct 30 13:32:59.526185 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:59.526034    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8f4bffb-6519-4c67-bb97-115403fafa6c-home\") pod \"auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc\" (UID: \"a8f4bffb-6519-4c67-bb97-115403fafa6c\") " pod="ai-tenant-e2e-shared-b-w1-29a3f3/auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc"
Oct 30 13:32:59.526343 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:59.526319    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8f4bffb-6519-4c67-bb97-115403fafa6c-model-cache\") pod \"auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc\" (UID: \"a8f4bffb-6519-4c67-bb97-115403fafa6c\") " pod="ai-tenant-e2e-shared-b-w1-29a3f3/auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc"
Oct 30 13:32:59.530207 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:59.528474    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8f4bffb-6519-4c67-bb97-115403fafa6c-dshm\") pod \"auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc\" (UID: \"a8f4bffb-6519-4c67-bb97-115403fafa6c\") " pod="ai-tenant-e2e-shared-b-w1-29a3f3/auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc"
Oct 30 13:32:59.530207 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:59.529585    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8f4bffb-6519-4c67-bb97-115403fafa6c-tls-certs\") pod \"auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc\" (UID: \"a8f4bffb-6519-4c67-bb97-115403fafa6c\") " pod="ai-tenant-e2e-shared-b-w1-29a3f3/auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc"
Oct 30 13:32:59.549455 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:59.549421    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlbtw\" (UniqueName: \"kubernetes.io/projected/a8f4bffb-6519-4c67-bb97-115403fafa6c-kube-api-access-qlbtw\") pod \"auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc\" (UID: \"a8f4bffb-6519-4c67-bb97-115403fafa6c\") " pod="ai-tenant-e2e-shared-b-w1-29a3f3/auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc"
Oct 30 13:32:59.728243 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:32:59.728206    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-29a3f3/auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc"
Oct 30 13:33:00.071901 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:33:00.071867    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-29a3f3/auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc"]
Oct 30 13:33:00.072888 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:33:00.072859    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda8f4bffb_6519_4c67_bb97_115403fafa6c.slice/crio-84d1cdaaec8804ded44a8574a56c0d368f001476e08850a89fa7b39040519d1e WatchSource:0}: Error finding container 84d1cdaaec8804ded44a8574a56c0d368f001476e08850a89fa7b39040519d1e: Status 404 returned error can't find the container with id 84d1cdaaec8804ded44a8574a56c0d368f001476e08850a89fa7b39040519d1e
Oct 30 13:33:00.215179 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:33:00.215095    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-29a3f3/auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc" event={"ID":"a8f4bffb-6519-4c67-bb97-115403fafa6c","Type":"ContainerStarted","Data":"7d78fcfb7f42c1f567c34ab6d5c9a07124650d5438379c87d5b3b41983ccf9e7"}
Oct 30 13:33:00.215179 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:33:00.215167    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-29a3f3/auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc" event={"ID":"a8f4bffb-6519-4c67-bb97-115403fafa6c","Type":"ContainerStarted","Data":"84d1cdaaec8804ded44a8574a56c0d368f001476e08850a89fa7b39040519d1e"}
Oct 30 13:33:00.322935 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:33:00.316698    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-29a3f3/auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc" podStartSLOduration=1.316671585 podStartE2EDuration="1.316671585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:32:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:33:00.276858543 +0000 UTC m=+1799.424125786" watchObservedRunningTime="2026-10-30 13:33:00.316671585 +0000 UTC m=+1799.463938827"
Oct 30 13:33:00.728646 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:33:00.728609    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-29a3f3/auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc"
Oct 30 13:33:01.388949 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:33:01.388918    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jdj4_f93ee017-ab79-447c-90e0-5b85b71dba24/console-operator/1.log"
Oct 30 13:33:01.390076 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:33:01.390038    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jdj4_f93ee017-ab79-447c-90e0-5b85b71dba24/console-operator/1.log"
Oct 30 13:33:01.395300 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:33:01.395276    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tnm_3a675ac7-fbe1-449b-accf-bb0456ed0bd3/ovn-acl-logging/0.log"
Oct 30 13:33:01.396298 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:33:01.396273    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tnm_3a675ac7-fbe1-449b-accf-bb0456ed0bd3/ovn-acl-logging/0.log"
Oct 30 13:33:09.729399 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:33:09.729352    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-29a3f3/auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc"
Oct 30 13:33:09.743424 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:33:09.743393    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-29a3f3/auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc"
Oct 30 13:33:10.313752 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:33:10.313717    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-29a3f3/auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc"
Oct 30 13:33:21.170795 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:33:21.170745    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:33:51.170023 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:33:51.169923    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:34:21.168584 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:34:21.168543    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:34:41.939587 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:34:41.939542    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq"]
Oct 30 13:34:41.940263 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:34:41.939902    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq" podUID="7f16ce08-b99b-4843-9213-9f78ac72626a" containerName="istio-proxy" containerID="cri-o://a82f07ff4230092f7b6607dbb5090ef02ae433e323b213a45628f6ec44fc3c84" gracePeriod=30
Oct 30 13:34:47.229634 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:34:47.229425    5093 generic.go:356] "Generic (PLEG): container finished" podID="7f16ce08-b99b-4843-9213-9f78ac72626a" containerID="a82f07ff4230092f7b6607dbb5090ef02ae433e323b213a45628f6ec44fc3c84" exitCode=0
Oct 30 13:34:47.229634 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:34:47.229468    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq" event={"ID":"7f16ce08-b99b-4843-9213-9f78ac72626a","Type":"ContainerDied","Data":"a82f07ff4230092f7b6607dbb5090ef02ae433e323b213a45628f6ec44fc3c84"}
Oct 30 13:34:47.278953 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:34:47.278909    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq"
Oct 30 13:34:47.438780 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:34:47.438739    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f16ce08-b99b-4843-9213-9f78ac72626a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f16ce08-b99b-4843-9213-9f78ac72626a-istiod-ca-cert\") pod \"7f16ce08-b99b-4843-9213-9f78ac72626a\" (UID: \"7f16ce08-b99b-4843-9213-9f78ac72626a\") "
Oct 30 13:34:47.438986 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:34:47.438819    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f16ce08-b99b-4843-9213-9f78ac72626a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f16ce08-b99b-4843-9213-9f78ac72626a-credential-socket\") pod \"7f16ce08-b99b-4843-9213-9f78ac72626a\" (UID: \"7f16ce08-b99b-4843-9213-9f78ac72626a\") "
Oct 30 13:34:47.438986 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:34:47.438914    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f16ce08-b99b-4843-9213-9f78ac72626a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f16ce08-b99b-4843-9213-9f78ac72626a-workload-socket\") pod \"7f16ce08-b99b-4843-9213-9f78ac72626a\" (UID: \"7f16ce08-b99b-4843-9213-9f78ac72626a\") "
Oct 30 13:34:47.438986 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:34:47.438960    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f16ce08-b99b-4843-9213-9f78ac72626a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f16ce08-b99b-4843-9213-9f78ac72626a-istio-podinfo\") pod \"7f16ce08-b99b-4843-9213-9f78ac72626a\" (UID: \"7f16ce08-b99b-4843-9213-9f78ac72626a\") "
Oct 30 13:34:47.439222 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:34:47.439026    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f16ce08-b99b-4843-9213-9f78ac72626a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f16ce08-b99b-4843-9213-9f78ac72626a-istio-data\") pod \"7f16ce08-b99b-4843-9213-9f78ac72626a\" (UID: \"7f16ce08-b99b-4843-9213-9f78ac72626a\") "
Oct 30 13:34:47.439222 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:34:47.439116    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f16ce08-b99b-4843-9213-9f78ac72626a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f16ce08-b99b-4843-9213-9f78ac72626a-workload-certs\") pod \"7f16ce08-b99b-4843-9213-9f78ac72626a\" (UID: \"7f16ce08-b99b-4843-9213-9f78ac72626a\") "
Oct 30 13:34:47.439346 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:34:47.439234    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f16ce08-b99b-4843-9213-9f78ac72626a-istio-token\" (UniqueName: \"kubernetes.io/projected/7f16ce08-b99b-4843-9213-9f78ac72626a-istio-token\") pod \"7f16ce08-b99b-4843-9213-9f78ac72626a\" (UID: \"7f16ce08-b99b-4843-9213-9f78ac72626a\") "
Oct 30 13:34:47.439409 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:34:47.439396    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f16ce08-b99b-4843-9213-9f78ac72626a-kube-api-access-94dnx\" (UniqueName: \"kubernetes.io/projected/7f16ce08-b99b-4843-9213-9f78ac72626a-kube-api-access-94dnx\") pod \"7f16ce08-b99b-4843-9213-9f78ac72626a\" (UID: \"7f16ce08-b99b-4843-9213-9f78ac72626a\") "
Oct 30 13:34:47.439516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:34:47.439501    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f16ce08-b99b-4843-9213-9f78ac72626a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f16ce08-b99b-4843-9213-9f78ac72626a-istio-envoy\") pod \"7f16ce08-b99b-4843-9213-9f78ac72626a\" (UID: \"7f16ce08-b99b-4843-9213-9f78ac72626a\") "
Oct 30 13:34:47.440540 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:34:47.440512    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f16ce08-b99b-4843-9213-9f78ac72626a-istiod-ca-cert" pod "7f16ce08-b99b-4843-9213-9f78ac72626a" (UID: "7f16ce08-b99b-4843-9213-9f78ac72626a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:34:47.440764 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:34:47.440746    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f16ce08-b99b-4843-9213-9f78ac72626a-credential-socket" pod "7f16ce08-b99b-4843-9213-9f78ac72626a" (UID: "7f16ce08-b99b-4843-9213-9f78ac72626a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:34:47.440928 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:34:47.440910    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f16ce08-b99b-4843-9213-9f78ac72626a-workload-socket" pod "7f16ce08-b99b-4843-9213-9f78ac72626a" (UID: "7f16ce08-b99b-4843-9213-9f78ac72626a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:34:47.443051 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:34:47.442988    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f16ce08-b99b-4843-9213-9f78ac72626a-workload-certs" pod "7f16ce08-b99b-4843-9213-9f78ac72626a" (UID: "7f16ce08-b99b-4843-9213-9f78ac72626a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:34:47.443268 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:34:47.443225    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f16ce08-b99b-4843-9213-9f78ac72626a-istio-data" pod "7f16ce08-b99b-4843-9213-9f78ac72626a" (UID: "7f16ce08-b99b-4843-9213-9f78ac72626a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:34:47.443810 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:34:47.443789    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f16ce08-b99b-4843-9213-9f78ac72626a-istio-podinfo" pod "7f16ce08-b99b-4843-9213-9f78ac72626a" (UID: "7f16ce08-b99b-4843-9213-9f78ac72626a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:34:47.445819 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:34:47.445778    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f16ce08-b99b-4843-9213-9f78ac72626a-istio-envoy" pod "7f16ce08-b99b-4843-9213-9f78ac72626a" (UID: "7f16ce08-b99b-4843-9213-9f78ac72626a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:34:47.454438 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:34:47.454382    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f16ce08-b99b-4843-9213-9f78ac72626a-kube-api-access-94dnx" pod "7f16ce08-b99b-4843-9213-9f78ac72626a" (UID: "7f16ce08-b99b-4843-9213-9f78ac72626a"). InnerVolumeSpecName "kube-api-access-94dnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:34:47.454986 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:34:47.454953    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f16ce08-b99b-4843-9213-9f78ac72626a-istio-token" pod "7f16ce08-b99b-4843-9213-9f78ac72626a" (UID: "7f16ce08-b99b-4843-9213-9f78ac72626a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:34:47.541650 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:34:47.541614    5093 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f16ce08-b99b-4843-9213-9f78ac72626a-istio-envoy\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:34:47.541650 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:34:47.541646    5093 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f16ce08-b99b-4843-9213-9f78ac72626a-istiod-ca-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:34:47.541650 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:34:47.541657    5093 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f16ce08-b99b-4843-9213-9f78ac72626a-credential-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:34:47.541917 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:34:47.541669    5093 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f16ce08-b99b-4843-9213-9f78ac72626a-workload-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:34:47.541917 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:34:47.541678    5093 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f16ce08-b99b-4843-9213-9f78ac72626a-istio-podinfo\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:34:47.541917 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:34:47.541686    5093 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f16ce08-b99b-4843-9213-9f78ac72626a-istio-data\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:34:47.541917 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:34:47.541694    5093 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f16ce08-b99b-4843-9213-9f78ac72626a-workload-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:34:47.541917 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:34:47.541705    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f16ce08-b99b-4843-9213-9f78ac72626a-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:34:47.541917 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:34:47.541713    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94dnx\" (UniqueName: \"kubernetes.io/projected/7f16ce08-b99b-4843-9213-9f78ac72626a-kube-api-access-94dnx\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:34:48.241967 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:34:48.241928    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq" event={"ID":"7f16ce08-b99b-4843-9213-9f78ac72626a","Type":"ContainerDied","Data":"4021fb81498c8f98199e7866fa13a992fef7d1e1104da33698d926a1b27a75e5"}
Oct 30 13:34:48.241967 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:34:48.241949    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq"
Oct 30 13:34:48.242580 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:34:48.242012    5093 scope.go:122] "RemoveContainer" containerID="a82f07ff4230092f7b6607dbb5090ef02ae433e323b213a45628f6ec44fc3c84"
Oct 30 13:34:48.312222 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:34:48.312187    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq"]
Oct 30 13:34:48.325550 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:34:48.325507    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-gbcrq"]
Oct 30 13:34:49.000668 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:34:49.000630    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f16ce08-b99b-4843-9213-9f78ac72626a" path="/var/lib/kubelet/pods/7f16ce08-b99b-4843-9213-9f78ac72626a/volumes"
Oct 30 13:34:49.832529 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:34:49.832489    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-c684d2/auth-test-model-c684d2-kserve-5cb979995c-6xppj"]
Oct 30 13:34:51.168963 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:34:51.168919    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:34:55.679783 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:34:55.679746    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-22af91/auth-test-model-22af91-kserve-84f6df67c7-hxtzx"]
Oct 30 13:35:04.856305 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:04.856245    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-c684d2/auth-test-model-c684d2-kserve-5cb979995c-6xppj" podUID="08f03e92-3e35-4a93-a240-f38ed6572622" containerName="main" containerID="cri-o://4a50304506e90af024cc0c350692c851745cd06d410023a9c746e4f8ea51b25a" gracePeriod=45
Oct 30 13:35:05.167851 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:05.167817    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-c684d2/auth-test-model-c684d2-kserve-5cb979995c-6xppj"
Oct 30 13:35:05.262625 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:05.262512    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08f03e92-3e35-4a93-a240-f38ed6572622-home\" (UniqueName: \"kubernetes.io/empty-dir/08f03e92-3e35-4a93-a240-f38ed6572622-home\") pod \"08f03e92-3e35-4a93-a240-f38ed6572622\" (UID: \"08f03e92-3e35-4a93-a240-f38ed6572622\") "
Oct 30 13:35:05.262625 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:05.262627    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08f03e92-3e35-4a93-a240-f38ed6572622-dshm\" (UniqueName: \"kubernetes.io/empty-dir/08f03e92-3e35-4a93-a240-f38ed6572622-dshm\") pod \"08f03e92-3e35-4a93-a240-f38ed6572622\" (UID: \"08f03e92-3e35-4a93-a240-f38ed6572622\") "
Oct 30 13:35:05.262894 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:05.262725    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08f03e92-3e35-4a93-a240-f38ed6572622-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08f03e92-3e35-4a93-a240-f38ed6572622-tmp-dir\") pod \"08f03e92-3e35-4a93-a240-f38ed6572622\" (UID: \"08f03e92-3e35-4a93-a240-f38ed6572622\") "
Oct 30 13:35:05.262894 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:05.262761    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08f03e92-3e35-4a93-a240-f38ed6572622-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08f03e92-3e35-4a93-a240-f38ed6572622-model-cache\") pod \"08f03e92-3e35-4a93-a240-f38ed6572622\" (UID: \"08f03e92-3e35-4a93-a240-f38ed6572622\") "
Oct 30 13:35:05.262894 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:05.262848    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f03e92-3e35-4a93-a240-f38ed6572622-home" pod "08f03e92-3e35-4a93-a240-f38ed6572622" (UID: "08f03e92-3e35-4a93-a240-f38ed6572622"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:35:05.263083 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:05.262902    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08f03e92-3e35-4a93-a240-f38ed6572622-tls-certs\" (UniqueName: \"kubernetes.io/secret/08f03e92-3e35-4a93-a240-f38ed6572622-tls-certs\") pod \"08f03e92-3e35-4a93-a240-f38ed6572622\" (UID: \"08f03e92-3e35-4a93-a240-f38ed6572622\") "
Oct 30 13:35:05.263083 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:05.262955    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08f03e92-3e35-4a93-a240-f38ed6572622-kube-api-access-d66hg\" (UniqueName: \"kubernetes.io/projected/08f03e92-3e35-4a93-a240-f38ed6572622-kube-api-access-d66hg\") pod \"08f03e92-3e35-4a93-a240-f38ed6572622\" (UID: \"08f03e92-3e35-4a93-a240-f38ed6572622\") "
Oct 30 13:35:05.263454 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:05.263423    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f03e92-3e35-4a93-a240-f38ed6572622-model-cache" pod "08f03e92-3e35-4a93-a240-f38ed6572622" (UID: "08f03e92-3e35-4a93-a240-f38ed6572622"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:35:05.263683 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:05.263661    5093 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08f03e92-3e35-4a93-a240-f38ed6572622-home\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:05.263776 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:05.263689    5093 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08f03e92-3e35-4a93-a240-f38ed6572622-model-cache\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:05.263926 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:05.263904    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f03e92-3e35-4a93-a240-f38ed6572622-tmp-dir" pod "08f03e92-3e35-4a93-a240-f38ed6572622" (UID: "08f03e92-3e35-4a93-a240-f38ed6572622"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:35:05.266291 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:05.266254    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08f03e92-3e35-4a93-a240-f38ed6572622-tls-certs" pod "08f03e92-3e35-4a93-a240-f38ed6572622" (UID: "08f03e92-3e35-4a93-a240-f38ed6572622"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:35:05.266534 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:05.266508    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f03e92-3e35-4a93-a240-f38ed6572622-dshm" pod "08f03e92-3e35-4a93-a240-f38ed6572622" (UID: "08f03e92-3e35-4a93-a240-f38ed6572622"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:35:05.267110 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:05.267085    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08f03e92-3e35-4a93-a240-f38ed6572622-kube-api-access-d66hg" pod "08f03e92-3e35-4a93-a240-f38ed6572622" (UID: "08f03e92-3e35-4a93-a240-f38ed6572622"). InnerVolumeSpecName "kube-api-access-d66hg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:35:05.365580 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:05.365538    5093 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08f03e92-3e35-4a93-a240-f38ed6572622-tmp-dir\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:05.365580 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:05.365580    5093 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08f03e92-3e35-4a93-a240-f38ed6572622-tls-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:05.365801 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:05.365597    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d66hg\" (UniqueName: \"kubernetes.io/projected/08f03e92-3e35-4a93-a240-f38ed6572622-kube-api-access-d66hg\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:05.365801 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:05.365613    5093 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08f03e92-3e35-4a93-a240-f38ed6572622-dshm\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:05.408002 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:05.407909    5093 generic.go:356] "Generic (PLEG): container finished" podID="08f03e92-3e35-4a93-a240-f38ed6572622" containerID="4a50304506e90af024cc0c350692c851745cd06d410023a9c746e4f8ea51b25a" exitCode=0
Oct 30 13:35:05.408002 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:05.407954    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-c684d2/auth-test-model-c684d2-kserve-5cb979995c-6xppj"
Oct 30 13:35:05.408002 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:05.407991    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-c684d2/auth-test-model-c684d2-kserve-5cb979995c-6xppj" event={"ID":"08f03e92-3e35-4a93-a240-f38ed6572622","Type":"ContainerDied","Data":"4a50304506e90af024cc0c350692c851745cd06d410023a9c746e4f8ea51b25a"}
Oct 30 13:35:05.408283 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:05.408040    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-c684d2/auth-test-model-c684d2-kserve-5cb979995c-6xppj" event={"ID":"08f03e92-3e35-4a93-a240-f38ed6572622","Type":"ContainerDied","Data":"5fbbf91632eb6c5eadd91f9c90d2aab9ca9883878926a127efaa7b1fd683183f"}
Oct 30 13:35:05.408283 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:05.408060    5093 scope.go:122] "RemoveContainer" containerID="4a50304506e90af024cc0c350692c851745cd06d410023a9c746e4f8ea51b25a"
Oct 30 13:35:05.420539 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:05.420520    5093 scope.go:122] "RemoveContainer" containerID="4a50304506e90af024cc0c350692c851745cd06d410023a9c746e4f8ea51b25a"
Oct 30 13:35:05.420838 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:35:05.420810    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a50304506e90af024cc0c350692c851745cd06d410023a9c746e4f8ea51b25a\": container with ID starting with 4a50304506e90af024cc0c350692c851745cd06d410023a9c746e4f8ea51b25a not found: ID does not exist" containerID="4a50304506e90af024cc0c350692c851745cd06d410023a9c746e4f8ea51b25a"
Oct 30 13:35:05.420946 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:05.420844    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a50304506e90af024cc0c350692c851745cd06d410023a9c746e4f8ea51b25a"} err="failed to get container status \"4a50304506e90af024cc0c350692c851745cd06d410023a9c746e4f8ea51b25a\": rpc error: code = NotFound desc = could not find container \"4a50304506e90af024cc0c350692c851745cd06d410023a9c746e4f8ea51b25a\": container with ID starting with 4a50304506e90af024cc0c350692c851745cd06d410023a9c746e4f8ea51b25a not found: ID does not exist"
Oct 30 13:35:05.468072 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:05.468031    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-c684d2/auth-test-model-c684d2-kserve-5cb979995c-6xppj"]
Oct 30 13:35:05.474465 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:05.474431    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-c684d2/auth-test-model-c684d2-kserve-5cb979995c-6xppj"]
Oct 30 13:35:07.002534 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:07.002489    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08f03e92-3e35-4a93-a240-f38ed6572622" path="/var/lib/kubelet/pods/08f03e92-3e35-4a93-a240-f38ed6572622/volumes"
Oct 30 13:35:09.591349 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:09.591313    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g"]
Oct 30 13:35:09.591773 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:09.591540    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g" podUID="911b7cb8-3eac-4576-be8b-3fecd0fea186" containerName="istio-proxy" containerID="cri-o://b91f94f69b5ab993381a08fcfcf0ad8f0745fc5627a050c0285a8160cd8ba288" gracePeriod=30
Oct 30 13:35:10.703994 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:10.703874    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-22af91/auth-test-model-22af91-kserve-84f6df67c7-hxtzx" podUID="21d2769d-dca2-4944-b72c-c18dc3a1e99d" containerName="main" containerID="cri-o://6feb364171de4f926887652a4d5ee2b0397cb08fc6010ce5d856093d675bc327" gracePeriod=45
Oct 30 13:35:11.017179 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:11.017099    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-22af91/auth-test-model-22af91-kserve-84f6df67c7-hxtzx"
Oct 30 13:35:11.188153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:11.187675    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21d2769d-dca2-4944-b72c-c18dc3a1e99d-home\" (UniqueName: \"kubernetes.io/empty-dir/21d2769d-dca2-4944-b72c-c18dc3a1e99d-home\") pod \"21d2769d-dca2-4944-b72c-c18dc3a1e99d\" (UID: \"21d2769d-dca2-4944-b72c-c18dc3a1e99d\") "
Oct 30 13:35:11.188153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:11.187860    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21d2769d-dca2-4944-b72c-c18dc3a1e99d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21d2769d-dca2-4944-b72c-c18dc3a1e99d-model-cache\") pod \"21d2769d-dca2-4944-b72c-c18dc3a1e99d\" (UID: \"21d2769d-dca2-4944-b72c-c18dc3a1e99d\") "
Oct 30 13:35:11.188153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:11.188008    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21d2769d-dca2-4944-b72c-c18dc3a1e99d-kube-api-access-hqj5g\" (UniqueName: \"kubernetes.io/projected/21d2769d-dca2-4944-b72c-c18dc3a1e99d-kube-api-access-hqj5g\") pod \"21d2769d-dca2-4944-b72c-c18dc3a1e99d\" (UID: \"21d2769d-dca2-4944-b72c-c18dc3a1e99d\") "
Oct 30 13:35:11.188153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:11.188152    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21d2769d-dca2-4944-b72c-c18dc3a1e99d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21d2769d-dca2-4944-b72c-c18dc3a1e99d-tmp-dir\") pod \"21d2769d-dca2-4944-b72c-c18dc3a1e99d\" (UID: \"21d2769d-dca2-4944-b72c-c18dc3a1e99d\") "
Oct 30 13:35:11.188539 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:11.188219    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21d2769d-dca2-4944-b72c-c18dc3a1e99d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/21d2769d-dca2-4944-b72c-c18dc3a1e99d-dshm\") pod \"21d2769d-dca2-4944-b72c-c18dc3a1e99d\" (UID: \"21d2769d-dca2-4944-b72c-c18dc3a1e99d\") "
Oct 30 13:35:11.188539 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:11.188272    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21d2769d-dca2-4944-b72c-c18dc3a1e99d-tls-certs\" (UniqueName: \"kubernetes.io/secret/21d2769d-dca2-4944-b72c-c18dc3a1e99d-tls-certs\") pod \"21d2769d-dca2-4944-b72c-c18dc3a1e99d\" (UID: \"21d2769d-dca2-4944-b72c-c18dc3a1e99d\") "
Oct 30 13:35:11.192751 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:11.190378    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21d2769d-dca2-4944-b72c-c18dc3a1e99d-home" pod "21d2769d-dca2-4944-b72c-c18dc3a1e99d" (UID: "21d2769d-dca2-4944-b72c-c18dc3a1e99d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:35:11.192751 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:11.190419    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21d2769d-dca2-4944-b72c-c18dc3a1e99d-model-cache" pod "21d2769d-dca2-4944-b72c-c18dc3a1e99d" (UID: "21d2769d-dca2-4944-b72c-c18dc3a1e99d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:35:11.192751 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:11.190557    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21d2769d-dca2-4944-b72c-c18dc3a1e99d-tmp-dir" pod "21d2769d-dca2-4944-b72c-c18dc3a1e99d" (UID: "21d2769d-dca2-4944-b72c-c18dc3a1e99d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:35:11.192751 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:11.192023    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21d2769d-dca2-4944-b72c-c18dc3a1e99d-tls-certs" pod "21d2769d-dca2-4944-b72c-c18dc3a1e99d" (UID: "21d2769d-dca2-4944-b72c-c18dc3a1e99d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:35:11.193157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:11.193103    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21d2769d-dca2-4944-b72c-c18dc3a1e99d-dshm" pod "21d2769d-dca2-4944-b72c-c18dc3a1e99d" (UID: "21d2769d-dca2-4944-b72c-c18dc3a1e99d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:35:11.193558 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:11.193533    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21d2769d-dca2-4944-b72c-c18dc3a1e99d-kube-api-access-hqj5g" pod "21d2769d-dca2-4944-b72c-c18dc3a1e99d" (UID: "21d2769d-dca2-4944-b72c-c18dc3a1e99d"). InnerVolumeSpecName "kube-api-access-hqj5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:35:11.290721 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:11.290622    5093 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21d2769d-dca2-4944-b72c-c18dc3a1e99d-model-cache\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:11.290721 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:11.290666    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqj5g\" (UniqueName: \"kubernetes.io/projected/21d2769d-dca2-4944-b72c-c18dc3a1e99d-kube-api-access-hqj5g\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:11.290721 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:11.290680    5093 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21d2769d-dca2-4944-b72c-c18dc3a1e99d-tmp-dir\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:11.290721 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:11.290691    5093 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21d2769d-dca2-4944-b72c-c18dc3a1e99d-dshm\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:11.290721 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:11.290707    5093 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21d2769d-dca2-4944-b72c-c18dc3a1e99d-tls-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:11.290721 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:11.290718    5093 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21d2769d-dca2-4944-b72c-c18dc3a1e99d-home\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:11.493154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:11.490796    5093 generic.go:356] "Generic (PLEG): container finished" podID="21d2769d-dca2-4944-b72c-c18dc3a1e99d" containerID="6feb364171de4f926887652a4d5ee2b0397cb08fc6010ce5d856093d675bc327" exitCode=0
Oct 30 13:35:11.493154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:11.490930    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-22af91/auth-test-model-22af91-kserve-84f6df67c7-hxtzx" event={"ID":"21d2769d-dca2-4944-b72c-c18dc3a1e99d","Type":"ContainerDied","Data":"6feb364171de4f926887652a4d5ee2b0397cb08fc6010ce5d856093d675bc327"}
Oct 30 13:35:11.493154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:11.490968    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-22af91/auth-test-model-22af91-kserve-84f6df67c7-hxtzx" event={"ID":"21d2769d-dca2-4944-b72c-c18dc3a1e99d","Type":"ContainerDied","Data":"422abcc1c99cc9ec5b0ef61861cda9342eb98fb0f1682eb73fdccb6d18be61d8"}
Oct 30 13:35:11.493154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:11.490990    5093 scope.go:122] "RemoveContainer" containerID="6feb364171de4f926887652a4d5ee2b0397cb08fc6010ce5d856093d675bc327"
Oct 30 13:35:11.493154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:11.491224    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-22af91/auth-test-model-22af91-kserve-84f6df67c7-hxtzx"
Oct 30 13:35:11.505929 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:11.505902    5093 scope.go:122] "RemoveContainer" containerID="6feb364171de4f926887652a4d5ee2b0397cb08fc6010ce5d856093d675bc327"
Oct 30 13:35:11.506262 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:35:11.506234    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6feb364171de4f926887652a4d5ee2b0397cb08fc6010ce5d856093d675bc327\": container with ID starting with 6feb364171de4f926887652a4d5ee2b0397cb08fc6010ce5d856093d675bc327 not found: ID does not exist" containerID="6feb364171de4f926887652a4d5ee2b0397cb08fc6010ce5d856093d675bc327"
Oct 30 13:35:11.506332 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:11.506271    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6feb364171de4f926887652a4d5ee2b0397cb08fc6010ce5d856093d675bc327"} err="failed to get container status \"6feb364171de4f926887652a4d5ee2b0397cb08fc6010ce5d856093d675bc327\": rpc error: code = NotFound desc = could not find container \"6feb364171de4f926887652a4d5ee2b0397cb08fc6010ce5d856093d675bc327\": container with ID starting with 6feb364171de4f926887652a4d5ee2b0397cb08fc6010ce5d856093d675bc327 not found: ID does not exist"
Oct 30 13:35:11.553290 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:11.553186    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-22af91/auth-test-model-22af91-kserve-84f6df67c7-hxtzx"]
Oct 30 13:35:11.565626 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:11.565591    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-22af91/auth-test-model-22af91-kserve-84f6df67c7-hxtzx"]
Oct 30 13:35:13.001542 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:13.001504    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21d2769d-dca2-4944-b72c-c18dc3a1e99d" path="/var/lib/kubelet/pods/21d2769d-dca2-4944-b72c-c18dc3a1e99d/volumes"
Oct 30 13:35:15.026636 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:15.026603    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g"
Oct 30 13:35:15.092474 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:15.092429    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/911b7cb8-3eac-4576-be8b-3fecd0fea186-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/911b7cb8-3eac-4576-be8b-3fecd0fea186-istiod-ca-cert\") pod \"911b7cb8-3eac-4576-be8b-3fecd0fea186\" (UID: \"911b7cb8-3eac-4576-be8b-3fecd0fea186\") "
Oct 30 13:35:15.093038 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:15.093001    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/911b7cb8-3eac-4576-be8b-3fecd0fea186-istiod-ca-cert" pod "911b7cb8-3eac-4576-be8b-3fecd0fea186" (UID: "911b7cb8-3eac-4576-be8b-3fecd0fea186"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:35:15.093688 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:15.093660    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/911b7cb8-3eac-4576-be8b-3fecd0fea186-istio-token\" (UniqueName: \"kubernetes.io/projected/911b7cb8-3eac-4576-be8b-3fecd0fea186-istio-token\") pod \"911b7cb8-3eac-4576-be8b-3fecd0fea186\" (UID: \"911b7cb8-3eac-4576-be8b-3fecd0fea186\") "
Oct 30 13:35:15.094878 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:15.094851    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/911b7cb8-3eac-4576-be8b-3fecd0fea186-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/911b7cb8-3eac-4576-be8b-3fecd0fea186-credential-socket\") pod \"911b7cb8-3eac-4576-be8b-3fecd0fea186\" (UID: \"911b7cb8-3eac-4576-be8b-3fecd0fea186\") "
Oct 30 13:35:15.094994 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:15.094922    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/911b7cb8-3eac-4576-be8b-3fecd0fea186-kube-api-access-27qvn\" (UniqueName: \"kubernetes.io/projected/911b7cb8-3eac-4576-be8b-3fecd0fea186-kube-api-access-27qvn\") pod \"911b7cb8-3eac-4576-be8b-3fecd0fea186\" (UID: \"911b7cb8-3eac-4576-be8b-3fecd0fea186\") "
Oct 30 13:35:15.094994 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:15.094951    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/911b7cb8-3eac-4576-be8b-3fecd0fea186-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/911b7cb8-3eac-4576-be8b-3fecd0fea186-istio-envoy\") pod \"911b7cb8-3eac-4576-be8b-3fecd0fea186\" (UID: \"911b7cb8-3eac-4576-be8b-3fecd0fea186\") "
Oct 30 13:35:15.095113 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:15.095045    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/911b7cb8-3eac-4576-be8b-3fecd0fea186-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/911b7cb8-3eac-4576-be8b-3fecd0fea186-workload-socket\") pod \"911b7cb8-3eac-4576-be8b-3fecd0fea186\" (UID: \"911b7cb8-3eac-4576-be8b-3fecd0fea186\") "
Oct 30 13:35:15.095113 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:15.095071    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/911b7cb8-3eac-4576-be8b-3fecd0fea186-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/911b7cb8-3eac-4576-be8b-3fecd0fea186-istio-podinfo\") pod \"911b7cb8-3eac-4576-be8b-3fecd0fea186\" (UID: \"911b7cb8-3eac-4576-be8b-3fecd0fea186\") "
Oct 30 13:35:15.095243 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:15.095115    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/911b7cb8-3eac-4576-be8b-3fecd0fea186-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/911b7cb8-3eac-4576-be8b-3fecd0fea186-workload-certs\") pod \"911b7cb8-3eac-4576-be8b-3fecd0fea186\" (UID: \"911b7cb8-3eac-4576-be8b-3fecd0fea186\") "
Oct 30 13:35:15.095243 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:15.095200    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/911b7cb8-3eac-4576-be8b-3fecd0fea186-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/911b7cb8-3eac-4576-be8b-3fecd0fea186-istio-data\") pod \"911b7cb8-3eac-4576-be8b-3fecd0fea186\" (UID: \"911b7cb8-3eac-4576-be8b-3fecd0fea186\") "
Oct 30 13:35:15.095911 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:15.095852    5093 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/911b7cb8-3eac-4576-be8b-3fecd0fea186-istiod-ca-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:15.096450 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:15.096425    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/911b7cb8-3eac-4576-be8b-3fecd0fea186-workload-certs" pod "911b7cb8-3eac-4576-be8b-3fecd0fea186" (UID: "911b7cb8-3eac-4576-be8b-3fecd0fea186"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:35:15.096665 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:15.096625    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/911b7cb8-3eac-4576-be8b-3fecd0fea186-credential-socket" pod "911b7cb8-3eac-4576-be8b-3fecd0fea186" (UID: "911b7cb8-3eac-4576-be8b-3fecd0fea186"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:35:15.096665 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:15.096639    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/911b7cb8-3eac-4576-be8b-3fecd0fea186-istio-data" pod "911b7cb8-3eac-4576-be8b-3fecd0fea186" (UID: "911b7cb8-3eac-4576-be8b-3fecd0fea186"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:35:15.097173 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:15.097094    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/911b7cb8-3eac-4576-be8b-3fecd0fea186-workload-socket" pod "911b7cb8-3eac-4576-be8b-3fecd0fea186" (UID: "911b7cb8-3eac-4576-be8b-3fecd0fea186"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:35:15.099853 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:15.098727    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/911b7cb8-3eac-4576-be8b-3fecd0fea186-istio-podinfo" pod "911b7cb8-3eac-4576-be8b-3fecd0fea186" (UID: "911b7cb8-3eac-4576-be8b-3fecd0fea186"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:35:15.099853 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:15.098774    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/911b7cb8-3eac-4576-be8b-3fecd0fea186-istio-envoy" pod "911b7cb8-3eac-4576-be8b-3fecd0fea186" (UID: "911b7cb8-3eac-4576-be8b-3fecd0fea186"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:35:15.100509 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:15.100474    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/911b7cb8-3eac-4576-be8b-3fecd0fea186-kube-api-access-27qvn" pod "911b7cb8-3eac-4576-be8b-3fecd0fea186" (UID: "911b7cb8-3eac-4576-be8b-3fecd0fea186"). InnerVolumeSpecName "kube-api-access-27qvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:35:15.104065 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:15.104038    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/911b7cb8-3eac-4576-be8b-3fecd0fea186-istio-token" pod "911b7cb8-3eac-4576-be8b-3fecd0fea186" (UID: "911b7cb8-3eac-4576-be8b-3fecd0fea186"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:35:15.198049 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:15.198001    5093 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/911b7cb8-3eac-4576-be8b-3fecd0fea186-credential-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:15.198049 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:15.198047    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27qvn\" (UniqueName: \"kubernetes.io/projected/911b7cb8-3eac-4576-be8b-3fecd0fea186-kube-api-access-27qvn\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:15.198490 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:15.198063    5093 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/911b7cb8-3eac-4576-be8b-3fecd0fea186-istio-envoy\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:15.198490 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:15.198077    5093 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/911b7cb8-3eac-4576-be8b-3fecd0fea186-workload-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:15.198490 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:15.198088    5093 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/911b7cb8-3eac-4576-be8b-3fecd0fea186-istio-podinfo\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:15.198490 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:15.198101    5093 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/911b7cb8-3eac-4576-be8b-3fecd0fea186-workload-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:15.198490 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:15.198113    5093 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/911b7cb8-3eac-4576-be8b-3fecd0fea186-istio-data\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:15.198490 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:15.198121    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/911b7cb8-3eac-4576-be8b-3fecd0fea186-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:15.527704 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:15.527605    5093 generic.go:356] "Generic (PLEG): container finished" podID="911b7cb8-3eac-4576-be8b-3fecd0fea186" containerID="b91f94f69b5ab993381a08fcfcf0ad8f0745fc5627a050c0285a8160cd8ba288" exitCode=0
Oct 30 13:35:15.527704 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:15.527666    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g"
Oct 30 13:35:15.527704 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:15.527680    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g" event={"ID":"911b7cb8-3eac-4576-be8b-3fecd0fea186","Type":"ContainerDied","Data":"b91f94f69b5ab993381a08fcfcf0ad8f0745fc5627a050c0285a8160cd8ba288"}
Oct 30 13:35:15.528000 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:15.527768    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g" event={"ID":"911b7cb8-3eac-4576-be8b-3fecd0fea186","Type":"ContainerDied","Data":"5ea11e8a9aac1d511e7ea004ed9c15edcd718fc97361dd3c6f9dee7e3d44e7f7"}
Oct 30 13:35:15.528000 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:15.527793    5093 scope.go:122] "RemoveContainer" containerID="b91f94f69b5ab993381a08fcfcf0ad8f0745fc5627a050c0285a8160cd8ba288"
Oct 30 13:35:15.543719 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:15.543694    5093 scope.go:122] "RemoveContainer" containerID="b91f94f69b5ab993381a08fcfcf0ad8f0745fc5627a050c0285a8160cd8ba288"
Oct 30 13:35:15.544203 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:35:15.544174    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b91f94f69b5ab993381a08fcfcf0ad8f0745fc5627a050c0285a8160cd8ba288\": container with ID starting with b91f94f69b5ab993381a08fcfcf0ad8f0745fc5627a050c0285a8160cd8ba288 not found: ID does not exist" containerID="b91f94f69b5ab993381a08fcfcf0ad8f0745fc5627a050c0285a8160cd8ba288"
Oct 30 13:35:15.544498 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:15.544460    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b91f94f69b5ab993381a08fcfcf0ad8f0745fc5627a050c0285a8160cd8ba288"} err="failed to get container status \"b91f94f69b5ab993381a08fcfcf0ad8f0745fc5627a050c0285a8160cd8ba288\": rpc error: code = NotFound desc = could not find container \"b91f94f69b5ab993381a08fcfcf0ad8f0745fc5627a050c0285a8160cd8ba288\": container with ID starting with b91f94f69b5ab993381a08fcfcf0ad8f0745fc5627a050c0285a8160cd8ba288 not found: ID does not exist"
Oct 30 13:35:15.588082 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:15.588041    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g"]
Oct 30 13:35:15.601196 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:15.601157    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-rqs9g"]
Oct 30 13:35:17.000966 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:17.000932    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="911b7cb8-3eac-4576-be8b-3fecd0fea186" path="/var/lib/kubelet/pods/911b7cb8-3eac-4576-be8b-3fecd0fea186/volumes"
Oct 30 13:35:21.168190 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:35:21.168147    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:35:37.175511 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:37.174679    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-f12b80/auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg"]
Oct 30 13:35:39.101739 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.101705    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp"]
Oct 30 13:35:39.102458 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.102432    5093 cpu_manager.go:414] "containerMap: removing container" podUID="911b7cb8-3eac-4576-be8b-3fecd0fea186" containerName="istio-proxy"
Oct 30 13:35:39.102458 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.102452    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="911b7cb8-3eac-4576-be8b-3fecd0fea186" containerName="istio-proxy"
Oct 30 13:35:39.102590 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.102488    5093 cpu_manager.go:414] "containerMap: removing container" podUID="7f16ce08-b99b-4843-9213-9f78ac72626a" containerName="istio-proxy"
Oct 30 13:35:39.102590 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.102494    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f16ce08-b99b-4843-9213-9f78ac72626a" containerName="istio-proxy"
Oct 30 13:35:39.102590 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.102506    5093 cpu_manager.go:414] "containerMap: removing container" podUID="08f03e92-3e35-4a93-a240-f38ed6572622" containerName="main"
Oct 30 13:35:39.102590 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.102511    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08f03e92-3e35-4a93-a240-f38ed6572622" containerName="main"
Oct 30 13:35:39.102590 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.102529    5093 cpu_manager.go:414] "containerMap: removing container" podUID="21d2769d-dca2-4944-b72c-c18dc3a1e99d" containerName="main"
Oct 30 13:35:39.102590 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.102535    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21d2769d-dca2-4944-b72c-c18dc3a1e99d" containerName="main"
Oct 30 13:35:39.102792 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.102661    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f16ce08-b99b-4843-9213-9f78ac72626a" containerName="istio-proxy"
Oct 30 13:35:39.102792 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.102670    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="08f03e92-3e35-4a93-a240-f38ed6572622" containerName="main"
Oct 30 13:35:39.102792 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.102678    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="21d2769d-dca2-4944-b72c-c18dc3a1e99d" containerName="main"
Oct 30 13:35:39.102792 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.102686    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="911b7cb8-3eac-4576-be8b-3fecd0fea186" containerName="istio-proxy"
Oct 30 13:35:39.114174 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.106668    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp"
Oct 30 13:35:39.136147 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.136085    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp"]
Oct 30 13:35:39.142612 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.142524    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-bd2fed-openshift-default-dockercfg-k26t7\""
Oct 30 13:35:39.163276 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.163207    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx"]
Oct 30 13:35:39.163493 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.163464    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx" podUID="67c19bdd-7c73-46e7-8e2b-58eef9290ce0" containerName="istio-proxy" containerID="cri-o://2dd023c3100090caf4c69c1ddb7616bfd617ff2af54ed1ee8f05d7c0224f088c" gracePeriod=30
Oct 30 13:35:39.175071 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.175003    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn"]
Oct 30 13:35:39.195389 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.195341    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-istiod-ca-cert\") pod \"e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp\" (UID: \"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e\") " pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp"
Oct 30 13:35:39.195569 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.195414    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-workload-certs\") pod \"e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp\" (UID: \"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e\") " pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp"
Oct 30 13:35:39.195569 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.195521    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-istio-podinfo\") pod \"e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp\" (UID: \"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e\") " pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp"
Oct 30 13:35:39.195569 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.195562    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-istio-token\") pod \"e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp\" (UID: \"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e\") " pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp"
Oct 30 13:35:39.195816 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.195662    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-istio-envoy\") pod \"e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp\" (UID: \"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e\") " pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp"
Oct 30 13:35:39.195816 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.195742    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-istio-data\") pod \"e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp\" (UID: \"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e\") " pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp"
Oct 30 13:35:39.195932 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.195880    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-credential-socket\") pod \"e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp\" (UID: \"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e\") " pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp"
Oct 30 13:35:39.195995 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.195938    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-workload-socket\") pod \"e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp\" (UID: \"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e\") " pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp"
Oct 30 13:35:39.195995 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.195978    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp4k2\" (UniqueName: \"kubernetes.io/projected/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-kube-api-access-xp4k2\") pod \"e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp\" (UID: \"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e\") " pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp"
Oct 30 13:35:39.217001 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.216246    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn" podUID="9980a626-91cc-492f-8918-1970311586e6" containerName="istio-proxy" containerID="cri-o://8cc739053c7972db5bf9dcc26c702a7eb70b3ef68338b1e2490bbe7773e73853" gracePeriod=30
Oct 30 13:35:39.297018 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.296976    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-credential-socket\") pod \"e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp\" (UID: \"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e\") " pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp"
Oct 30 13:35:39.297225 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.297043    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-workload-socket\") pod \"e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp\" (UID: \"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e\") " pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp"
Oct 30 13:35:39.297225 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.297068    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp4k2\" (UniqueName: \"kubernetes.io/projected/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-kube-api-access-xp4k2\") pod \"e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp\" (UID: \"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e\") " pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp"
Oct 30 13:35:39.297225 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.297168    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-istiod-ca-cert\") pod \"e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp\" (UID: \"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e\") " pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp"
Oct 30 13:35:39.297225 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.297198    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-workload-certs\") pod \"e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp\" (UID: \"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e\") " pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp"
Oct 30 13:35:39.297460 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.297303    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-istio-podinfo\") pod \"e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp\" (UID: \"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e\") " pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp"
Oct 30 13:35:39.297531 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.297449    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-istio-token\") pod \"e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp\" (UID: \"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e\") " pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp"
Oct 30 13:35:39.297531 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.297495    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-credential-socket\") pod \"e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp\" (UID: \"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e\") " pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp"
Oct 30 13:35:39.297646 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.297539    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-workload-socket\") pod \"e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp\" (UID: \"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e\") " pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp"
Oct 30 13:35:39.297646 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.297587    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-istio-envoy\") pod \"e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp\" (UID: \"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e\") " pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp"
Oct 30 13:35:39.297828 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.297666    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-istio-data\") pod \"e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp\" (UID: \"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e\") " pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp"
Oct 30 13:35:39.297888 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.297852    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-workload-certs\") pod \"e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp\" (UID: \"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e\") " pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp"
Oct 30 13:35:39.298078 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.298054    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-istiod-ca-cert\") pod \"e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp\" (UID: \"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e\") " pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp"
Oct 30 13:35:39.298277 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.298118    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-istio-data\") pod \"e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp\" (UID: \"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e\") " pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp"
Oct 30 13:35:39.300574 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.300548    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-istio-envoy\") pod \"e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp\" (UID: \"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e\") " pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp"
Oct 30 13:35:39.300888 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.300865    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-istio-podinfo\") pod \"e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp\" (UID: \"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e\") " pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp"
Oct 30 13:35:39.322223 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.320215    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp4k2\" (UniqueName: \"kubernetes.io/projected/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-kube-api-access-xp4k2\") pod \"e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp\" (UID: \"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e\") " pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp"
Oct 30 13:35:39.322478 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.322454    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-istio-token\") pod \"e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp\" (UID: \"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e\") " pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp"
Oct 30 13:35:39.473155 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.472044    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp"
Oct 30 13:35:39.852683 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.852649    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp"]
Oct 30 13:35:39.855629 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:35:39.855593    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18b2b0a1_af31_4cb1_acdf_6357fbf0ed7e.slice/crio-54ace99a523e8155f8f4a62fd88f48f4640e73defc78e0e8f0719c96532e287a WatchSource:0}: Error finding container 54ace99a523e8155f8f4a62fd88f48f4640e73defc78e0e8f0719c96532e287a: Status 404 returned error can't find the container with id 54ace99a523e8155f8f4a62fd88f48f4640e73defc78e0e8f0719c96532e287a
Oct 30 13:35:39.859180 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.859085    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:35:39.859315 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.859204    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:35:39.859315 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:39.859252    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:35:40.843157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:40.842248    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp" event={"ID":"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e","Type":"ContainerStarted","Data":"bf678e91aa9af56a31ab09de6b7287b69bc50de05e9cf7030530a2bf694faf47"}
Oct 30 13:35:40.843157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:40.842323    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp" event={"ID":"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e","Type":"ContainerStarted","Data":"54ace99a523e8155f8f4a62fd88f48f4640e73defc78e0e8f0719c96532e287a"}
Oct 30 13:35:40.914023 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:40.913948    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp" podStartSLOduration=1.913925469 podStartE2EDuration="1.913925469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:35:40.899764089 +0000 UTC m=+1960.047031330" watchObservedRunningTime="2026-10-30 13:35:40.913925469 +0000 UTC m=+1960.061192711"
Oct 30 13:35:41.473200 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:41.473162    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp"
Oct 30 13:35:41.480468 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:41.480435    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp"
Oct 30 13:35:41.851742 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:41.851651    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp"
Oct 30 13:35:41.852638 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:41.852616    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp"
Oct 30 13:35:42.137616 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:42.137536    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5"]
Oct 30 13:35:42.858695 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:42.858653    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5" podUID="b57be897-da31-47b5-bf14-5a11969f94d0" containerName="istio-proxy" containerID="cri-o://8e8edc539375a5e71330d33108ebf64346727cb1db108d9c180d4ab554c1fcc2" gracePeriod=30
Oct 30 13:35:43.311376 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:43.311340    5093 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:35:43.311597 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:43.311408    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn" podUID="9980a626-91cc-492f-8918-1970311586e6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:35:44.537870 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.537838    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx"
Oct 30 13:35:44.592400 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.592375    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn"
Oct 30 13:35:44.644904 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.644861    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-workload-socket\") pod \"67c19bdd-7c73-46e7-8e2b-58eef9290ce0\" (UID: \"67c19bdd-7c73-46e7-8e2b-58eef9290ce0\") "
Oct 30 13:35:44.645097 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.644936    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9980a626-91cc-492f-8918-1970311586e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9980a626-91cc-492f-8918-1970311586e6-workload-socket\") pod \"9980a626-91cc-492f-8918-1970311586e6\" (UID: \"9980a626-91cc-492f-8918-1970311586e6\") "
Oct 30 13:35:44.645097 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.645000    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-istio-podinfo\") pod \"67c19bdd-7c73-46e7-8e2b-58eef9290ce0\" (UID: \"67c19bdd-7c73-46e7-8e2b-58eef9290ce0\") "
Oct 30 13:35:44.645097 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.645047    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-workload-certs\") pod \"67c19bdd-7c73-46e7-8e2b-58eef9290ce0\" (UID: \"67c19bdd-7c73-46e7-8e2b-58eef9290ce0\") "
Oct 30 13:35:44.645296 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.645099    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9980a626-91cc-492f-8918-1970311586e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9980a626-91cc-492f-8918-1970311586e6-istio-envoy\") pod \"9980a626-91cc-492f-8918-1970311586e6\" (UID: \"9980a626-91cc-492f-8918-1970311586e6\") "
Oct 30 13:35:44.645296 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.645188    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9980a626-91cc-492f-8918-1970311586e6-kube-api-access-xc6s2\" (UniqueName: \"kubernetes.io/projected/9980a626-91cc-492f-8918-1970311586e6-kube-api-access-xc6s2\") pod \"9980a626-91cc-492f-8918-1970311586e6\" (UID: \"9980a626-91cc-492f-8918-1970311586e6\") "
Oct 30 13:35:44.645296 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.645220    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-istiod-ca-cert\") pod \"67c19bdd-7c73-46e7-8e2b-58eef9290ce0\" (UID: \"67c19bdd-7c73-46e7-8e2b-58eef9290ce0\") "
Oct 30 13:35:44.645296 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.645285    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-istio-data\") pod \"67c19bdd-7c73-46e7-8e2b-58eef9290ce0\" (UID: \"67c19bdd-7c73-46e7-8e2b-58eef9290ce0\") "
Oct 30 13:35:44.645510 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.645372    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-istio-envoy\") pod \"67c19bdd-7c73-46e7-8e2b-58eef9290ce0\" (UID: \"67c19bdd-7c73-46e7-8e2b-58eef9290ce0\") "
Oct 30 13:35:44.645510 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.645478    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9980a626-91cc-492f-8918-1970311586e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9980a626-91cc-492f-8918-1970311586e6-istio-podinfo\") pod \"9980a626-91cc-492f-8918-1970311586e6\" (UID: \"9980a626-91cc-492f-8918-1970311586e6\") "
Oct 30 13:35:44.645613 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.645506    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-credential-socket\") pod \"67c19bdd-7c73-46e7-8e2b-58eef9290ce0\" (UID: \"67c19bdd-7c73-46e7-8e2b-58eef9290ce0\") "
Oct 30 13:35:44.645613 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.645555    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-kube-api-access-sdcbl\" (UniqueName: \"kubernetes.io/projected/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-kube-api-access-sdcbl\") pod \"67c19bdd-7c73-46e7-8e2b-58eef9290ce0\" (UID: \"67c19bdd-7c73-46e7-8e2b-58eef9290ce0\") "
Oct 30 13:35:44.645613 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.645597    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9980a626-91cc-492f-8918-1970311586e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9980a626-91cc-492f-8918-1970311586e6-workload-certs\") pod \"9980a626-91cc-492f-8918-1970311586e6\" (UID: \"9980a626-91cc-492f-8918-1970311586e6\") "
Oct 30 13:35:44.645766 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.645642    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9980a626-91cc-492f-8918-1970311586e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9980a626-91cc-492f-8918-1970311586e6-credential-socket\") pod \"9980a626-91cc-492f-8918-1970311586e6\" (UID: \"9980a626-91cc-492f-8918-1970311586e6\") "
Oct 30 13:35:44.645766 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.645684    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9980a626-91cc-492f-8918-1970311586e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9980a626-91cc-492f-8918-1970311586e6-istiod-ca-cert\") pod \"9980a626-91cc-492f-8918-1970311586e6\" (UID: \"9980a626-91cc-492f-8918-1970311586e6\") "
Oct 30 13:35:44.645766 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.645715    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9980a626-91cc-492f-8918-1970311586e6-istio-token\" (UniqueName: \"kubernetes.io/projected/9980a626-91cc-492f-8918-1970311586e6-istio-token\") pod \"9980a626-91cc-492f-8918-1970311586e6\" (UID: \"9980a626-91cc-492f-8918-1970311586e6\") "
Oct 30 13:35:44.645766 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.645743    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-istio-token\" (UniqueName: \"kubernetes.io/projected/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-istio-token\") pod \"67c19bdd-7c73-46e7-8e2b-58eef9290ce0\" (UID: \"67c19bdd-7c73-46e7-8e2b-58eef9290ce0\") "
Oct 30 13:35:44.645972 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.645843    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9980a626-91cc-492f-8918-1970311586e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9980a626-91cc-492f-8918-1970311586e6-istio-data\") pod \"9980a626-91cc-492f-8918-1970311586e6\" (UID: \"9980a626-91cc-492f-8918-1970311586e6\") "
Oct 30 13:35:44.646834 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.646798    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9980a626-91cc-492f-8918-1970311586e6-istio-data" pod "9980a626-91cc-492f-8918-1970311586e6" (UID: "9980a626-91cc-492f-8918-1970311586e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:35:44.650868 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.650151    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-workload-certs" pod "67c19bdd-7c73-46e7-8e2b-58eef9290ce0" (UID: "67c19bdd-7c73-46e7-8e2b-58eef9290ce0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:35:44.650868 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.650362    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-istio-data" pod "67c19bdd-7c73-46e7-8e2b-58eef9290ce0" (UID: "67c19bdd-7c73-46e7-8e2b-58eef9290ce0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:35:44.650868 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.650633    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-istiod-ca-cert" pod "67c19bdd-7c73-46e7-8e2b-58eef9290ce0" (UID: "67c19bdd-7c73-46e7-8e2b-58eef9290ce0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:35:44.651663 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.651637    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9980a626-91cc-492f-8918-1970311586e6-workload-socket" pod "9980a626-91cc-492f-8918-1970311586e6" (UID: "9980a626-91cc-492f-8918-1970311586e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:35:44.651798 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.651744    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-workload-socket" pod "67c19bdd-7c73-46e7-8e2b-58eef9290ce0" (UID: "67c19bdd-7c73-46e7-8e2b-58eef9290ce0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:35:44.651983 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.651958    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9980a626-91cc-492f-8918-1970311586e6-credential-socket" pod "9980a626-91cc-492f-8918-1970311586e6" (UID: "9980a626-91cc-492f-8918-1970311586e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:35:44.652174 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.652154    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9980a626-91cc-492f-8918-1970311586e6-workload-certs" pod "9980a626-91cc-492f-8918-1970311586e6" (UID: "9980a626-91cc-492f-8918-1970311586e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:35:44.655695 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.655667    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9980a626-91cc-492f-8918-1970311586e6-istiod-ca-cert" pod "9980a626-91cc-492f-8918-1970311586e6" (UID: "9980a626-91cc-492f-8918-1970311586e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:35:44.656671 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.656640    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-credential-socket" pod "67c19bdd-7c73-46e7-8e2b-58eef9290ce0" (UID: "67c19bdd-7c73-46e7-8e2b-58eef9290ce0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:35:44.668674 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.668635    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-kube-api-access-sdcbl" pod "67c19bdd-7c73-46e7-8e2b-58eef9290ce0" (UID: "67c19bdd-7c73-46e7-8e2b-58eef9290ce0"). InnerVolumeSpecName "kube-api-access-sdcbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:35:44.668853 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.668810    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9980a626-91cc-492f-8918-1970311586e6-istio-token" pod "9980a626-91cc-492f-8918-1970311586e6" (UID: "9980a626-91cc-492f-8918-1970311586e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:35:44.669233 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.669208    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9980a626-91cc-492f-8918-1970311586e6-istio-podinfo" pod "9980a626-91cc-492f-8918-1970311586e6" (UID: "9980a626-91cc-492f-8918-1970311586e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:35:44.669322 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.669248    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9980a626-91cc-492f-8918-1970311586e6-istio-envoy" pod "9980a626-91cc-492f-8918-1970311586e6" (UID: "9980a626-91cc-492f-8918-1970311586e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:35:44.669362 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.669322    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-istio-envoy" pod "67c19bdd-7c73-46e7-8e2b-58eef9290ce0" (UID: "67c19bdd-7c73-46e7-8e2b-58eef9290ce0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:35:44.669362 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.669157    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-istio-podinfo" pod "67c19bdd-7c73-46e7-8e2b-58eef9290ce0" (UID: "67c19bdd-7c73-46e7-8e2b-58eef9290ce0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:35:44.669362 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.669324    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9980a626-91cc-492f-8918-1970311586e6-kube-api-access-xc6s2" pod "9980a626-91cc-492f-8918-1970311586e6" (UID: "9980a626-91cc-492f-8918-1970311586e6"). InnerVolumeSpecName "kube-api-access-xc6s2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:35:44.671225 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.671199    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-istio-token" pod "67c19bdd-7c73-46e7-8e2b-58eef9290ce0" (UID: "67c19bdd-7c73-46e7-8e2b-58eef9290ce0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:35:44.748656 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.748557    5093 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9980a626-91cc-492f-8918-1970311586e6-istio-podinfo\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:44.748656 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.748600    5093 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-credential-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:44.748656 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.748615    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdcbl\" (UniqueName: \"kubernetes.io/projected/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-kube-api-access-sdcbl\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:44.748656 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.748630    5093 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9980a626-91cc-492f-8918-1970311586e6-workload-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:44.748656 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.748644    5093 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9980a626-91cc-492f-8918-1970311586e6-credential-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:44.748656 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.748656    5093 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9980a626-91cc-492f-8918-1970311586e6-istiod-ca-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:44.749078 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.748672    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9980a626-91cc-492f-8918-1970311586e6-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:44.749078 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.748684    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:44.749078 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.748696    5093 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9980a626-91cc-492f-8918-1970311586e6-istio-data\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:44.749078 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.748710    5093 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-workload-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:44.749078 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.748722    5093 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9980a626-91cc-492f-8918-1970311586e6-workload-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:44.749078 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.748736    5093 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-istio-podinfo\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:44.749078 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.748749    5093 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-workload-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:44.749078 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.748761    5093 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9980a626-91cc-492f-8918-1970311586e6-istio-envoy\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:44.749078 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.748774    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc6s2\" (UniqueName: \"kubernetes.io/projected/9980a626-91cc-492f-8918-1970311586e6-kube-api-access-xc6s2\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:44.749078 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.748788    5093 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-istiod-ca-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:44.749078 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.748800    5093 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-istio-data\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:44.749078 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.748811    5093 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67c19bdd-7c73-46e7-8e2b-58eef9290ce0-istio-envoy\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:44.880691 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.880653    5093 generic.go:356] "Generic (PLEG): container finished" podID="9980a626-91cc-492f-8918-1970311586e6" containerID="8cc739053c7972db5bf9dcc26c702a7eb70b3ef68338b1e2490bbe7773e73853" exitCode=0
Oct 30 13:35:44.880849 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.880705    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn"
Oct 30 13:35:44.880849 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.880801    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn" event={"ID":"9980a626-91cc-492f-8918-1970311586e6","Type":"ContainerDied","Data":"8cc739053c7972db5bf9dcc26c702a7eb70b3ef68338b1e2490bbe7773e73853"}
Oct 30 13:35:44.881009 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.880865    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn" event={"ID":"9980a626-91cc-492f-8918-1970311586e6","Type":"ContainerDied","Data":"e29378a23617c2c525a4aec3b6dc9272395ededf5a17025b209fb0d7bbb37f20"}
Oct 30 13:35:44.881009 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.880894    5093 scope.go:122] "RemoveContainer" containerID="8cc739053c7972db5bf9dcc26c702a7eb70b3ef68338b1e2490bbe7773e73853"
Oct 30 13:35:44.882766 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.882633    5093 generic.go:356] "Generic (PLEG): container finished" podID="67c19bdd-7c73-46e7-8e2b-58eef9290ce0" containerID="2dd023c3100090caf4c69c1ddb7616bfd617ff2af54ed1ee8f05d7c0224f088c" exitCode=0
Oct 30 13:35:44.882766 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.882664    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx" event={"ID":"67c19bdd-7c73-46e7-8e2b-58eef9290ce0","Type":"ContainerDied","Data":"2dd023c3100090caf4c69c1ddb7616bfd617ff2af54ed1ee8f05d7c0224f088c"}
Oct 30 13:35:44.882766 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.882693    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx"
Oct 30 13:35:44.882766 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.882713    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx" event={"ID":"67c19bdd-7c73-46e7-8e2b-58eef9290ce0","Type":"ContainerDied","Data":"8015f3c8762822c688c963ba227d9c58b9314af0c08527d78c5b00f930a64536"}
Oct 30 13:35:44.894819 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.894477    5093 scope.go:122] "RemoveContainer" containerID="8cc739053c7972db5bf9dcc26c702a7eb70b3ef68338b1e2490bbe7773e73853"
Oct 30 13:35:44.895113 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:35:44.895081    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cc739053c7972db5bf9dcc26c702a7eb70b3ef68338b1e2490bbe7773e73853\": container with ID starting with 8cc739053c7972db5bf9dcc26c702a7eb70b3ef68338b1e2490bbe7773e73853 not found: ID does not exist" containerID="8cc739053c7972db5bf9dcc26c702a7eb70b3ef68338b1e2490bbe7773e73853"
Oct 30 13:35:44.895270 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.895139    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cc739053c7972db5bf9dcc26c702a7eb70b3ef68338b1e2490bbe7773e73853"} err="failed to get container status \"8cc739053c7972db5bf9dcc26c702a7eb70b3ef68338b1e2490bbe7773e73853\": rpc error: code = NotFound desc = could not find container \"8cc739053c7972db5bf9dcc26c702a7eb70b3ef68338b1e2490bbe7773e73853\": container with ID starting with 8cc739053c7972db5bf9dcc26c702a7eb70b3ef68338b1e2490bbe7773e73853 not found: ID does not exist"
Oct 30 13:35:44.895270 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.895169    5093 scope.go:122] "RemoveContainer" containerID="2dd023c3100090caf4c69c1ddb7616bfd617ff2af54ed1ee8f05d7c0224f088c"
Oct 30 13:35:44.911446 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.911425    5093 scope.go:122] "RemoveContainer" containerID="2dd023c3100090caf4c69c1ddb7616bfd617ff2af54ed1ee8f05d7c0224f088c"
Oct 30 13:35:44.911745 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:35:44.911722    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dd023c3100090caf4c69c1ddb7616bfd617ff2af54ed1ee8f05d7c0224f088c\": container with ID starting with 2dd023c3100090caf4c69c1ddb7616bfd617ff2af54ed1ee8f05d7c0224f088c not found: ID does not exist" containerID="2dd023c3100090caf4c69c1ddb7616bfd617ff2af54ed1ee8f05d7c0224f088c"
Oct 30 13:35:44.911819 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.911761    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dd023c3100090caf4c69c1ddb7616bfd617ff2af54ed1ee8f05d7c0224f088c"} err="failed to get container status \"2dd023c3100090caf4c69c1ddb7616bfd617ff2af54ed1ee8f05d7c0224f088c\": rpc error: code = NotFound desc = could not find container \"2dd023c3100090caf4c69c1ddb7616bfd617ff2af54ed1ee8f05d7c0224f088c\": container with ID starting with 2dd023c3100090caf4c69c1ddb7616bfd617ff2af54ed1ee8f05d7c0224f088c not found: ID does not exist"
Oct 30 13:35:44.956604 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.956570    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx"]
Oct 30 13:35:44.970470 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:44.970432    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qwljx"]
Oct 30 13:35:45.005638 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:45.005561    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67c19bdd-7c73-46e7-8e2b-58eef9290ce0" path="/var/lib/kubelet/pods/67c19bdd-7c73-46e7-8e2b-58eef9290ce0/volumes"
Oct 30 13:35:45.005938 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:45.005922    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn"]
Oct 30 13:35:45.005987 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:45.005956    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-rl4pn"]
Oct 30 13:35:45.212507 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:45.212472    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-bd2fed-68ff7bd4b6-d9tt2"]
Oct 30 13:35:45.213285 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:45.213260    5093 cpu_manager.go:414] "containerMap: removing container" podUID="67c19bdd-7c73-46e7-8e2b-58eef9290ce0" containerName="istio-proxy"
Oct 30 13:35:45.213285 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:45.213284    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67c19bdd-7c73-46e7-8e2b-58eef9290ce0" containerName="istio-proxy"
Oct 30 13:35:45.213465 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:45.213313    5093 cpu_manager.go:414] "containerMap: removing container" podUID="9980a626-91cc-492f-8918-1970311586e6" containerName="istio-proxy"
Oct 30 13:35:45.213465 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:45.213321    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9980a626-91cc-492f-8918-1970311586e6" containerName="istio-proxy"
Oct 30 13:35:45.213913 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:45.213512    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="9980a626-91cc-492f-8918-1970311586e6" containerName="istio-proxy"
Oct 30 13:35:45.213913 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:45.213540    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="67c19bdd-7c73-46e7-8e2b-58eef9290ce0" containerName="istio-proxy"
Oct 30 13:35:45.219122 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:45.219089    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-bd2fed-68ff7bd4b6-d9tt2"
Oct 30 13:35:45.235820 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:45.235768    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-bd2fed\""
Oct 30 13:35:45.243862 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:45.243830    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-bd2fed-68ff7bd4b6-d9tt2"]
Oct 30 13:35:45.262919 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:45.262833    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64b19bcc-57c8-4838-a94c-5b918d4da2e2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-bd2fed-68ff7bd4b6-d9tt2\" (UID: \"64b19bcc-57c8-4838-a94c-5b918d4da2e2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-bd2fed-68ff7bd4b6-d9tt2"
Oct 30 13:35:45.263116 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:45.263094    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b9r7\" (UniqueName: \"kubernetes.io/projected/64b19bcc-57c8-4838-a94c-5b918d4da2e2-kube-api-access-4b9r7\") pod \"payload-processing-e2e-extmod-a-bd2fed-68ff7bd4b6-d9tt2\" (UID: \"64b19bcc-57c8-4838-a94c-5b918d4da2e2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-bd2fed-68ff7bd4b6-d9tt2"
Oct 30 13:35:45.279799 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:45.279767    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-bd2fed-dockercfg-6wtw9\""
Oct 30 13:35:45.395153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:45.393202    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64b19bcc-57c8-4838-a94c-5b918d4da2e2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-bd2fed-68ff7bd4b6-d9tt2\" (UID: \"64b19bcc-57c8-4838-a94c-5b918d4da2e2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-bd2fed-68ff7bd4b6-d9tt2"
Oct 30 13:35:45.395153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:45.393490    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b9r7\" (UniqueName: \"kubernetes.io/projected/64b19bcc-57c8-4838-a94c-5b918d4da2e2-kube-api-access-4b9r7\") pod \"payload-processing-e2e-extmod-a-bd2fed-68ff7bd4b6-d9tt2\" (UID: \"64b19bcc-57c8-4838-a94c-5b918d4da2e2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-bd2fed-68ff7bd4b6-d9tt2"
Oct 30 13:35:45.395153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:45.394778    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64b19bcc-57c8-4838-a94c-5b918d4da2e2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-bd2fed-68ff7bd4b6-d9tt2\" (UID: \"64b19bcc-57c8-4838-a94c-5b918d4da2e2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-bd2fed-68ff7bd4b6-d9tt2"
Oct 30 13:35:45.442166 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:45.437964    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b9r7\" (UniqueName: \"kubernetes.io/projected/64b19bcc-57c8-4838-a94c-5b918d4da2e2-kube-api-access-4b9r7\") pod \"payload-processing-e2e-extmod-a-bd2fed-68ff7bd4b6-d9tt2\" (UID: \"64b19bcc-57c8-4838-a94c-5b918d4da2e2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-bd2fed-68ff7bd4b6-d9tt2"
Oct 30 13:35:45.559157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:45.559047    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-bd2fed-68ff7bd4b6-d9tt2"
Oct 30 13:35:45.873295 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:35:45.873260    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64b19bcc_57c8_4838_a94c_5b918d4da2e2.slice/crio-99d8b332d357ab6a771d5d5835814116e9ad9b4571cd4259fe9a73b22eb2a329 WatchSource:0}: Error finding container 99d8b332d357ab6a771d5d5835814116e9ad9b4571cd4259fe9a73b22eb2a329: Status 404 returned error can't find the container with id 99d8b332d357ab6a771d5d5835814116e9ad9b4571cd4259fe9a73b22eb2a329
Oct 30 13:35:45.874454 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:45.874425    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-bd2fed-68ff7bd4b6-d9tt2"]
Oct 30 13:35:45.902693 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:45.902659    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-bd2fed-68ff7bd4b6-d9tt2" event={"ID":"64b19bcc-57c8-4838-a94c-5b918d4da2e2","Type":"ContainerStarted","Data":"99d8b332d357ab6a771d5d5835814116e9ad9b4571cd4259fe9a73b22eb2a329"}
Oct 30 13:35:46.913381 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:46.913341    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-bd2fed-68ff7bd4b6-d9tt2" event={"ID":"64b19bcc-57c8-4838-a94c-5b918d4da2e2","Type":"ContainerStarted","Data":"8a527cd176c5ede8c33363022a30d2134b4b009b09d12b4fa5e36779313431da"}
Oct 30 13:35:46.913876 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:46.913404    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-bd2fed-68ff7bd4b6-d9tt2"
Oct 30 13:35:46.959885 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:46.959819    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-bd2fed-68ff7bd4b6-d9tt2" podStartSLOduration=1.9598004900000001 podStartE2EDuration="1.95980049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:35:46.956390935 +0000 UTC m=+1966.103658177" watchObservedRunningTime="2026-10-30 13:35:46.95980049 +0000 UTC m=+1966.107067738"
Oct 30 13:35:47.000482 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:47.000451    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9980a626-91cc-492f-8918-1970311586e6" path="/var/lib/kubelet/pods/9980a626-91cc-492f-8918-1970311586e6/volumes"
Oct 30 13:35:48.162731 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.162695    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5"
Oct 30 13:35:48.220541 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.220494    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-310546-6db5dd49d6-xfhp4"]
Oct 30 13:35:48.221364 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.221337    5093 cpu_manager.go:414] "containerMap: removing container" podUID="b57be897-da31-47b5-bf14-5a11969f94d0" containerName="istio-proxy"
Oct 30 13:35:48.221364 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.221361    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b57be897-da31-47b5-bf14-5a11969f94d0" containerName="istio-proxy"
Oct 30 13:35:48.221588 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.221573    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="b57be897-da31-47b5-bf14-5a11969f94d0" containerName="istio-proxy"
Oct 30 13:35:48.225739 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.225709    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-310546-6db5dd49d6-xfhp4"
Oct 30 13:35:48.247220 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.233949    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b57be897-da31-47b5-bf14-5a11969f94d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b57be897-da31-47b5-bf14-5a11969f94d0-istio-podinfo\") pod \"b57be897-da31-47b5-bf14-5a11969f94d0\" (UID: \"b57be897-da31-47b5-bf14-5a11969f94d0\") "
Oct 30 13:35:48.247220 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.234074    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b57be897-da31-47b5-bf14-5a11969f94d0-istio-token\" (UniqueName: \"kubernetes.io/projected/b57be897-da31-47b5-bf14-5a11969f94d0-istio-token\") pod \"b57be897-da31-47b5-bf14-5a11969f94d0\" (UID: \"b57be897-da31-47b5-bf14-5a11969f94d0\") "
Oct 30 13:35:48.247220 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.234179    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b57be897-da31-47b5-bf14-5a11969f94d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b57be897-da31-47b5-bf14-5a11969f94d0-workload-socket\") pod \"b57be897-da31-47b5-bf14-5a11969f94d0\" (UID: \"b57be897-da31-47b5-bf14-5a11969f94d0\") "
Oct 30 13:35:48.247220 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.234215    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b57be897-da31-47b5-bf14-5a11969f94d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b57be897-da31-47b5-bf14-5a11969f94d0-istio-envoy\") pod \"b57be897-da31-47b5-bf14-5a11969f94d0\" (UID: \"b57be897-da31-47b5-bf14-5a11969f94d0\") "
Oct 30 13:35:48.247220 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.234244    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b57be897-da31-47b5-bf14-5a11969f94d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b57be897-da31-47b5-bf14-5a11969f94d0-istiod-ca-cert\") pod \"b57be897-da31-47b5-bf14-5a11969f94d0\" (UID: \"b57be897-da31-47b5-bf14-5a11969f94d0\") "
Oct 30 13:35:48.247220 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.234267    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b57be897-da31-47b5-bf14-5a11969f94d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b57be897-da31-47b5-bf14-5a11969f94d0-credential-socket\") pod \"b57be897-da31-47b5-bf14-5a11969f94d0\" (UID: \"b57be897-da31-47b5-bf14-5a11969f94d0\") "
Oct 30 13:35:48.247220 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.234293    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b57be897-da31-47b5-bf14-5a11969f94d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b57be897-da31-47b5-bf14-5a11969f94d0-workload-certs\") pod \"b57be897-da31-47b5-bf14-5a11969f94d0\" (UID: \"b57be897-da31-47b5-bf14-5a11969f94d0\") "
Oct 30 13:35:48.247220 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.234355    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b57be897-da31-47b5-bf14-5a11969f94d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b57be897-da31-47b5-bf14-5a11969f94d0-istio-data\") pod \"b57be897-da31-47b5-bf14-5a11969f94d0\" (UID: \"b57be897-da31-47b5-bf14-5a11969f94d0\") "
Oct 30 13:35:48.247220 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.234418    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b57be897-da31-47b5-bf14-5a11969f94d0-kube-api-access-bwdd5\" (UniqueName: \"kubernetes.io/projected/b57be897-da31-47b5-bf14-5a11969f94d0-kube-api-access-bwdd5\") pod \"b57be897-da31-47b5-bf14-5a11969f94d0\" (UID: \"b57be897-da31-47b5-bf14-5a11969f94d0\") "
Oct 30 13:35:48.247220 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.236923    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b57be897-da31-47b5-bf14-5a11969f94d0-workload-certs" pod "b57be897-da31-47b5-bf14-5a11969f94d0" (UID: "b57be897-da31-47b5-bf14-5a11969f94d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:35:48.247220 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.236940    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b57be897-da31-47b5-bf14-5a11969f94d0-istiod-ca-cert" pod "b57be897-da31-47b5-bf14-5a11969f94d0" (UID: "b57be897-da31-47b5-bf14-5a11969f94d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:35:48.247220 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.237281    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b57be897-da31-47b5-bf14-5a11969f94d0-workload-socket" pod "b57be897-da31-47b5-bf14-5a11969f94d0" (UID: "b57be897-da31-47b5-bf14-5a11969f94d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:35:48.247220 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.237420    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b57be897-da31-47b5-bf14-5a11969f94d0-credential-socket" pod "b57be897-da31-47b5-bf14-5a11969f94d0" (UID: "b57be897-da31-47b5-bf14-5a11969f94d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:35:48.247220 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.237479    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b57be897-da31-47b5-bf14-5a11969f94d0-istio-data" pod "b57be897-da31-47b5-bf14-5a11969f94d0" (UID: "b57be897-da31-47b5-bf14-5a11969f94d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:35:48.247220 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.238117    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b57be897-da31-47b5-bf14-5a11969f94d0-kube-api-access-bwdd5" pod "b57be897-da31-47b5-bf14-5a11969f94d0" (UID: "b57be897-da31-47b5-bf14-5a11969f94d0"). InnerVolumeSpecName "kube-api-access-bwdd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:35:48.247220 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.241364    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b57be897-da31-47b5-bf14-5a11969f94d0-istio-token" pod "b57be897-da31-47b5-bf14-5a11969f94d0" (UID: "b57be897-da31-47b5-bf14-5a11969f94d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:35:48.247220 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.243060    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b57be897-da31-47b5-bf14-5a11969f94d0-istio-podinfo" pod "b57be897-da31-47b5-bf14-5a11969f94d0" (UID: "b57be897-da31-47b5-bf14-5a11969f94d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:35:48.247220 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.243254    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b57be897-da31-47b5-bf14-5a11969f94d0-istio-envoy" pod "b57be897-da31-47b5-bf14-5a11969f94d0" (UID: "b57be897-da31-47b5-bf14-5a11969f94d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:35:48.247220 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.244079    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-310546\""
Oct 30 13:35:48.247220 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.244249    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-310546-dockercfg-z8czf\""
Oct 30 13:35:48.247220 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.244412    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-310546-6db5dd49d6-xfhp4"]
Oct 30 13:35:48.283648 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.283616    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-310546-7c9f444fc9-xw4r4"]
Oct 30 13:35:48.287930 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.287907    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-310546-7c9f444fc9-xw4r4"
Oct 30 13:35:48.304884 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.304849    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-310546-7c9f444fc9-xw4r4"]
Oct 30 13:35:48.337063 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.337017    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/958c7031-f31a-4c34-9dd3-d01530719385-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-310546-6db5dd49d6-xfhp4\" (UID: \"958c7031-f31a-4c34-9dd3-d01530719385\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-310546-6db5dd49d6-xfhp4"
Oct 30 13:35:48.337319 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.337122    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whvrq\" (UniqueName: \"kubernetes.io/projected/958c7031-f31a-4c34-9dd3-d01530719385-kube-api-access-whvrq\") pod \"payload-processing-e2e-extmod-a-310546-6db5dd49d6-xfhp4\" (UID: \"958c7031-f31a-4c34-9dd3-d01530719385\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-310546-6db5dd49d6-xfhp4"
Oct 30 13:35:48.337483 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.337461    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwdd5\" (UniqueName: \"kubernetes.io/projected/b57be897-da31-47b5-bf14-5a11969f94d0-kube-api-access-bwdd5\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:48.337483 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.337483    5093 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b57be897-da31-47b5-bf14-5a11969f94d0-istio-podinfo\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:48.337693 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.337498    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b57be897-da31-47b5-bf14-5a11969f94d0-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:48.337693 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.337511    5093 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b57be897-da31-47b5-bf14-5a11969f94d0-workload-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:48.337693 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.337522    5093 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b57be897-da31-47b5-bf14-5a11969f94d0-istio-envoy\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:48.337693 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.337533    5093 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b57be897-da31-47b5-bf14-5a11969f94d0-istiod-ca-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:48.337693 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.337546    5093 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b57be897-da31-47b5-bf14-5a11969f94d0-credential-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:48.337693 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.337561    5093 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b57be897-da31-47b5-bf14-5a11969f94d0-workload-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:48.337693 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.337574    5093 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b57be897-da31-47b5-bf14-5a11969f94d0-istio-data\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:48.447154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.446934    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45ths\" (UniqueName: \"kubernetes.io/projected/63b21cf6-7b3b-48b6-88a1-d03e804d33fa-kube-api-access-45ths\") pod \"payload-pre-processing-e2e-extmod-a-310546-7c9f444fc9-xw4r4\" (UID: \"63b21cf6-7b3b-48b6-88a1-d03e804d33fa\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-310546-7c9f444fc9-xw4r4"
Oct 30 13:35:48.447154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.447071    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63b21cf6-7b3b-48b6-88a1-d03e804d33fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-310546-7c9f444fc9-xw4r4\" (UID: \"63b21cf6-7b3b-48b6-88a1-d03e804d33fa\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-310546-7c9f444fc9-xw4r4"
Oct 30 13:35:48.447434 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.447326    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/958c7031-f31a-4c34-9dd3-d01530719385-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-310546-6db5dd49d6-xfhp4\" (UID: \"958c7031-f31a-4c34-9dd3-d01530719385\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-310546-6db5dd49d6-xfhp4"
Oct 30 13:35:48.447434 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.447397    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whvrq\" (UniqueName: \"kubernetes.io/projected/958c7031-f31a-4c34-9dd3-d01530719385-kube-api-access-whvrq\") pod \"payload-processing-e2e-extmod-a-310546-6db5dd49d6-xfhp4\" (UID: \"958c7031-f31a-4c34-9dd3-d01530719385\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-310546-6db5dd49d6-xfhp4"
Oct 30 13:35:48.451169 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.448429    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/958c7031-f31a-4c34-9dd3-d01530719385-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-310546-6db5dd49d6-xfhp4\" (UID: \"958c7031-f31a-4c34-9dd3-d01530719385\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-310546-6db5dd49d6-xfhp4"
Oct 30 13:35:48.476967 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.473086    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whvrq\" (UniqueName: \"kubernetes.io/projected/958c7031-f31a-4c34-9dd3-d01530719385-kube-api-access-whvrq\") pod \"payload-processing-e2e-extmod-a-310546-6db5dd49d6-xfhp4\" (UID: \"958c7031-f31a-4c34-9dd3-d01530719385\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-310546-6db5dd49d6-xfhp4"
Oct 30 13:35:48.549356 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.549305    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63b21cf6-7b3b-48b6-88a1-d03e804d33fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-310546-7c9f444fc9-xw4r4\" (UID: \"63b21cf6-7b3b-48b6-88a1-d03e804d33fa\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-310546-7c9f444fc9-xw4r4"
Oct 30 13:35:48.549628 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.549611    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45ths\" (UniqueName: \"kubernetes.io/projected/63b21cf6-7b3b-48b6-88a1-d03e804d33fa-kube-api-access-45ths\") pod \"payload-pre-processing-e2e-extmod-a-310546-7c9f444fc9-xw4r4\" (UID: \"63b21cf6-7b3b-48b6-88a1-d03e804d33fa\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-310546-7c9f444fc9-xw4r4"
Oct 30 13:35:48.550146 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.550111    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63b21cf6-7b3b-48b6-88a1-d03e804d33fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-310546-7c9f444fc9-xw4r4\" (UID: \"63b21cf6-7b3b-48b6-88a1-d03e804d33fa\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-310546-7c9f444fc9-xw4r4"
Oct 30 13:35:48.570628 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.570597    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45ths\" (UniqueName: \"kubernetes.io/projected/63b21cf6-7b3b-48b6-88a1-d03e804d33fa-kube-api-access-45ths\") pod \"payload-pre-processing-e2e-extmod-a-310546-7c9f444fc9-xw4r4\" (UID: \"63b21cf6-7b3b-48b6-88a1-d03e804d33fa\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-310546-7c9f444fc9-xw4r4"
Oct 30 13:35:48.585462 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.585434    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-310546-6db5dd49d6-xfhp4"
Oct 30 13:35:48.609782 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.609740    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-310546-7c9f444fc9-xw4r4"
Oct 30 13:35:48.903182 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:35:48.903143    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod958c7031_f31a_4c34_9dd3_d01530719385.slice/crio-0e3f0c98a7acaea5933b92cd2d4860ad89b0bbfd6e59c4e9956204b43ce02c07 WatchSource:0}: Error finding container 0e3f0c98a7acaea5933b92cd2d4860ad89b0bbfd6e59c4e9956204b43ce02c07: Status 404 returned error can't find the container with id 0e3f0c98a7acaea5933b92cd2d4860ad89b0bbfd6e59c4e9956204b43ce02c07
Oct 30 13:35:48.903532 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.903505    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-310546-6db5dd49d6-xfhp4"]
Oct 30 13:35:48.930872 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.930846    5093 generic.go:356] "Generic (PLEG): container finished" podID="b57be897-da31-47b5-bf14-5a11969f94d0" containerID="8e8edc539375a5e71330d33108ebf64346727cb1db108d9c180d4ab554c1fcc2" exitCode=0
Oct 30 13:35:48.930988 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.930908    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5"
Oct 30 13:35:48.930988 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.930931    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5" event={"ID":"b57be897-da31-47b5-bf14-5a11969f94d0","Type":"ContainerDied","Data":"8e8edc539375a5e71330d33108ebf64346727cb1db108d9c180d4ab554c1fcc2"}
Oct 30 13:35:48.931112 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.931003    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5" event={"ID":"b57be897-da31-47b5-bf14-5a11969f94d0","Type":"ContainerDied","Data":"3137bba8025fcafbce26be2425fe09ed94d7e9b109fb2a11bc69b08cd36791bc"}
Oct 30 13:35:48.931112 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.931043    5093 scope.go:122] "RemoveContainer" containerID="8e8edc539375a5e71330d33108ebf64346727cb1db108d9c180d4ab554c1fcc2"
Oct 30 13:35:48.932871 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.932847    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-310546-6db5dd49d6-xfhp4" event={"ID":"958c7031-f31a-4c34-9dd3-d01530719385","Type":"ContainerStarted","Data":"0e3f0c98a7acaea5933b92cd2d4860ad89b0bbfd6e59c4e9956204b43ce02c07"}
Oct 30 13:35:48.941283 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.941258    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-310546-7c9f444fc9-xw4r4"]
Oct 30 13:35:48.942342 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:35:48.942319    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63b21cf6_7b3b_48b6_88a1_d03e804d33fa.slice/crio-5a68d294b32223b74100fd5fd81b71e01924b88348abfcd9a644daa0e2098856 WatchSource:0}: Error finding container 5a68d294b32223b74100fd5fd81b71e01924b88348abfcd9a644daa0e2098856: Status 404 returned error can't find the container with id 5a68d294b32223b74100fd5fd81b71e01924b88348abfcd9a644daa0e2098856
Oct 30 13:35:48.977728 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.977697    5093 scope.go:122] "RemoveContainer" containerID="8e8edc539375a5e71330d33108ebf64346727cb1db108d9c180d4ab554c1fcc2"
Oct 30 13:35:48.978159 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:35:48.978120    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e8edc539375a5e71330d33108ebf64346727cb1db108d9c180d4ab554c1fcc2\": container with ID starting with 8e8edc539375a5e71330d33108ebf64346727cb1db108d9c180d4ab554c1fcc2 not found: ID does not exist" containerID="8e8edc539375a5e71330d33108ebf64346727cb1db108d9c180d4ab554c1fcc2"
Oct 30 13:35:48.978273 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:48.978172    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e8edc539375a5e71330d33108ebf64346727cb1db108d9c180d4ab554c1fcc2"} err="failed to get container status \"8e8edc539375a5e71330d33108ebf64346727cb1db108d9c180d4ab554c1fcc2\": rpc error: code = NotFound desc = could not find container \"8e8edc539375a5e71330d33108ebf64346727cb1db108d9c180d4ab554c1fcc2\": container with ID starting with 8e8edc539375a5e71330d33108ebf64346727cb1db108d9c180d4ab554c1fcc2 not found: ID does not exist"
Oct 30 13:35:49.019755 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:49.019723    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5"]
Oct 30 13:35:49.023848 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:49.023820    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-5wtg5"]
Oct 30 13:35:49.945465 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:49.945370    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-310546-7c9f444fc9-xw4r4" event={"ID":"63b21cf6-7b3b-48b6-88a1-d03e804d33fa","Type":"ContainerStarted","Data":"b4a90681e85564f7bb07f692e80acd481e000126e36818af5e23963dd4df4ff5"}
Oct 30 13:35:49.945465 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:49.945459    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-310546-7c9f444fc9-xw4r4"
Oct 30 13:35:49.945991 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:49.945488    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-310546-7c9f444fc9-xw4r4" event={"ID":"63b21cf6-7b3b-48b6-88a1-d03e804d33fa","Type":"ContainerStarted","Data":"5a68d294b32223b74100fd5fd81b71e01924b88348abfcd9a644daa0e2098856"}
Oct 30 13:35:49.947114 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:49.947086    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-310546-6db5dd49d6-xfhp4" event={"ID":"958c7031-f31a-4c34-9dd3-d01530719385","Type":"ContainerStarted","Data":"8d81c0d9f185a07d983112478f4398e2ebc3ac4e9dc7dfb340f9423b802d1590"}
Oct 30 13:35:49.947823 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:49.947802    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-310546-6db5dd49d6-xfhp4"
Oct 30 13:35:49.986682 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:49.986625    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-310546-7c9f444fc9-xw4r4" podStartSLOduration=1.986609968 podStartE2EDuration="1.986609968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:35:49.985112311 +0000 UTC m=+1969.132379553" watchObservedRunningTime="2026-10-30 13:35:49.986609968 +0000 UTC m=+1969.133877208"
Oct 30 13:35:50.028749 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:50.028107    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-310546-6db5dd49d6-xfhp4" podStartSLOduration=2.028092895 podStartE2EDuration="2.028092895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:35:50.026077667 +0000 UTC m=+1969.173344911" watchObservedRunningTime="2026-10-30 13:35:50.028092895 +0000 UTC m=+1969.175360169"
Oct 30 13:35:51.000253 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:51.000211    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b57be897-da31-47b5-bf14-5a11969f94d0" path="/var/lib/kubelet/pods/b57be897-da31-47b5-bf14-5a11969f94d0/volumes"
Oct 30 13:35:51.172397 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:35:51.172355    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:35:52.207263 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:52.207218    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-f12b80/auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg" podUID="98a0175c-d89f-44c3-aae9-4585f86974f8" containerName="main" containerID="cri-o://157abe299b737087a241aa5a703bc2e47cb63315377aafd05a2a3c512dec4882" gracePeriod=45
Oct 30 13:35:52.504300 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:52.504272    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-f12b80/auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg"
Oct 30 13:35:52.608483 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:52.608447    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98a0175c-d89f-44c3-aae9-4585f86974f8-tls-certs\" (UniqueName: \"kubernetes.io/secret/98a0175c-d89f-44c3-aae9-4585f86974f8-tls-certs\") pod \"98a0175c-d89f-44c3-aae9-4585f86974f8\" (UID: \"98a0175c-d89f-44c3-aae9-4585f86974f8\") "
Oct 30 13:35:52.608688 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:52.608602    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98a0175c-d89f-44c3-aae9-4585f86974f8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/98a0175c-d89f-44c3-aae9-4585f86974f8-dshm\") pod \"98a0175c-d89f-44c3-aae9-4585f86974f8\" (UID: \"98a0175c-d89f-44c3-aae9-4585f86974f8\") "
Oct 30 13:35:52.608688 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:52.608675    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98a0175c-d89f-44c3-aae9-4585f86974f8-home\" (UniqueName: \"kubernetes.io/empty-dir/98a0175c-d89f-44c3-aae9-4585f86974f8-home\") pod \"98a0175c-d89f-44c3-aae9-4585f86974f8\" (UID: \"98a0175c-d89f-44c3-aae9-4585f86974f8\") "
Oct 30 13:35:52.608809 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:52.608710    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98a0175c-d89f-44c3-aae9-4585f86974f8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98a0175c-d89f-44c3-aae9-4585f86974f8-model-cache\") pod \"98a0175c-d89f-44c3-aae9-4585f86974f8\" (UID: \"98a0175c-d89f-44c3-aae9-4585f86974f8\") "
Oct 30 13:35:52.608809 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:52.608750    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98a0175c-d89f-44c3-aae9-4585f86974f8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98a0175c-d89f-44c3-aae9-4585f86974f8-tmp-dir\") pod \"98a0175c-d89f-44c3-aae9-4585f86974f8\" (UID: \"98a0175c-d89f-44c3-aae9-4585f86974f8\") "
Oct 30 13:35:52.608916 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:52.608843    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98a0175c-d89f-44c3-aae9-4585f86974f8-kube-api-access-8vr8m\" (UniqueName: \"kubernetes.io/projected/98a0175c-d89f-44c3-aae9-4585f86974f8-kube-api-access-8vr8m\") pod \"98a0175c-d89f-44c3-aae9-4585f86974f8\" (UID: \"98a0175c-d89f-44c3-aae9-4585f86974f8\") "
Oct 30 13:35:52.608965 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:52.608935    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98a0175c-d89f-44c3-aae9-4585f86974f8-home" pod "98a0175c-d89f-44c3-aae9-4585f86974f8" (UID: "98a0175c-d89f-44c3-aae9-4585f86974f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:35:52.609175 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:52.609145    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98a0175c-d89f-44c3-aae9-4585f86974f8-tmp-dir" pod "98a0175c-d89f-44c3-aae9-4585f86974f8" (UID: "98a0175c-d89f-44c3-aae9-4585f86974f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:35:52.609306 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:52.609203    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98a0175c-d89f-44c3-aae9-4585f86974f8-model-cache" pod "98a0175c-d89f-44c3-aae9-4585f86974f8" (UID: "98a0175c-d89f-44c3-aae9-4585f86974f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:35:52.609462 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:52.609447    5093 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98a0175c-d89f-44c3-aae9-4585f86974f8-home\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:52.609527 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:52.609464    5093 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98a0175c-d89f-44c3-aae9-4585f86974f8-model-cache\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:52.609527 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:52.609472    5093 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98a0175c-d89f-44c3-aae9-4585f86974f8-tmp-dir\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:52.612574 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:52.612543    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98a0175c-d89f-44c3-aae9-4585f86974f8-dshm" pod "98a0175c-d89f-44c3-aae9-4585f86974f8" (UID: "98a0175c-d89f-44c3-aae9-4585f86974f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:35:52.612676 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:52.612576    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98a0175c-d89f-44c3-aae9-4585f86974f8-kube-api-access-8vr8m" pod "98a0175c-d89f-44c3-aae9-4585f86974f8" (UID: "98a0175c-d89f-44c3-aae9-4585f86974f8"). InnerVolumeSpecName "kube-api-access-8vr8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:35:52.612676 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:52.612602    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98a0175c-d89f-44c3-aae9-4585f86974f8-tls-certs" pod "98a0175c-d89f-44c3-aae9-4585f86974f8" (UID: "98a0175c-d89f-44c3-aae9-4585f86974f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:35:52.710817 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:52.710784    5093 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98a0175c-d89f-44c3-aae9-4585f86974f8-dshm\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:52.710817 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:52.710815    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vr8m\" (UniqueName: \"kubernetes.io/projected/98a0175c-d89f-44c3-aae9-4585f86974f8-kube-api-access-8vr8m\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:52.711020 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:52.710826    5093 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98a0175c-d89f-44c3-aae9-4585f86974f8-tls-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:35:52.982566 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:52.982532    5093 generic.go:356] "Generic (PLEG): container finished" podID="98a0175c-d89f-44c3-aae9-4585f86974f8" containerID="157abe299b737087a241aa5a703bc2e47cb63315377aafd05a2a3c512dec4882" exitCode=0
Oct 30 13:35:52.982766 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:52.982579    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-f12b80/auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg"
Oct 30 13:35:52.982766 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:52.982610    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-f12b80/auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg" event={"ID":"98a0175c-d89f-44c3-aae9-4585f86974f8","Type":"ContainerDied","Data":"157abe299b737087a241aa5a703bc2e47cb63315377aafd05a2a3c512dec4882"}
Oct 30 13:35:52.982766 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:52.982644    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-f12b80/auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg" event={"ID":"98a0175c-d89f-44c3-aae9-4585f86974f8","Type":"ContainerDied","Data":"0eb65abd551ef6f7222ca322fd4ceb5a5e337911eab528a9b8edd274c575a0c6"}
Oct 30 13:35:52.982766 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:52.982660    5093 scope.go:122] "RemoveContainer" containerID="157abe299b737087a241aa5a703bc2e47cb63315377aafd05a2a3c512dec4882"
Oct 30 13:35:53.000234 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:53.000197    5093 scope.go:122] "RemoveContainer" containerID="157abe299b737087a241aa5a703bc2e47cb63315377aafd05a2a3c512dec4882"
Oct 30 13:35:53.000586 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:35:53.000562    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"157abe299b737087a241aa5a703bc2e47cb63315377aafd05a2a3c512dec4882\": container with ID starting with 157abe299b737087a241aa5a703bc2e47cb63315377aafd05a2a3c512dec4882 not found: ID does not exist" containerID="157abe299b737087a241aa5a703bc2e47cb63315377aafd05a2a3c512dec4882"
Oct 30 13:35:53.000655 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:53.000601    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"157abe299b737087a241aa5a703bc2e47cb63315377aafd05a2a3c512dec4882"} err="failed to get container status \"157abe299b737087a241aa5a703bc2e47cb63315377aafd05a2a3c512dec4882\": rpc error: code = NotFound desc = could not find container \"157abe299b737087a241aa5a703bc2e47cb63315377aafd05a2a3c512dec4882\": container with ID starting with 157abe299b737087a241aa5a703bc2e47cb63315377aafd05a2a3c512dec4882 not found: ID does not exist"
Oct 30 13:35:53.040722 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:53.040677    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-f12b80/auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg"]
Oct 30 13:35:53.053255 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:53.053213    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-f12b80/auth-test-model-f12b80-kserve-7789c9d7bf-6wvjg"]
Oct 30 13:35:55.002774 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:55.002738    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98a0175c-d89f-44c3-aae9-4585f86974f8" path="/var/lib/kubelet/pods/98a0175c-d89f-44c3-aae9-4585f86974f8/volumes"
Oct 30 13:35:57.433861 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:57.433818    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp"]
Oct 30 13:35:57.434919 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:57.434895    5093 cpu_manager.go:414] "containerMap: removing container" podUID="98a0175c-d89f-44c3-aae9-4585f86974f8" containerName="main"
Oct 30 13:35:57.434989 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:57.434922    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98a0175c-d89f-44c3-aae9-4585f86974f8" containerName="main"
Oct 30 13:35:57.435089 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:57.435071    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="98a0175c-d89f-44c3-aae9-4585f86974f8" containerName="main"
Oct 30 13:35:57.440865 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:57.440836    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp"
Oct 30 13:35:57.460699 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:57.460656    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp"]
Oct 30 13:35:57.468733 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:57.468701    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-51dc9a-openshift-default-dockercfg-dgxr2\""
Oct 30 13:35:57.494246 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:57.494196    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8285d6ec-8ad7-4a33-a696-d32242da2f80-istiod-ca-cert\") pod \"e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp\" (UID: \"8285d6ec-8ad7-4a33-a696-d32242da2f80\") " pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp"
Oct 30 13:35:57.494246 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:57.494255    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8285d6ec-8ad7-4a33-a696-d32242da2f80-istio-data\") pod \"e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp\" (UID: \"8285d6ec-8ad7-4a33-a696-d32242da2f80\") " pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp"
Oct 30 13:35:57.494559 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:57.494444    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c8pm\" (UniqueName: \"kubernetes.io/projected/8285d6ec-8ad7-4a33-a696-d32242da2f80-kube-api-access-2c8pm\") pod \"e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp\" (UID: \"8285d6ec-8ad7-4a33-a696-d32242da2f80\") " pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp"
Oct 30 13:35:57.494559 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:57.494516    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8285d6ec-8ad7-4a33-a696-d32242da2f80-istio-podinfo\") pod \"e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp\" (UID: \"8285d6ec-8ad7-4a33-a696-d32242da2f80\") " pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp"
Oct 30 13:35:57.494817 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:57.494785    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8285d6ec-8ad7-4a33-a696-d32242da2f80-credential-socket\") pod \"e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp\" (UID: \"8285d6ec-8ad7-4a33-a696-d32242da2f80\") " pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp"
Oct 30 13:35:57.494945 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:57.494850    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8285d6ec-8ad7-4a33-a696-d32242da2f80-workload-socket\") pod \"e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp\" (UID: \"8285d6ec-8ad7-4a33-a696-d32242da2f80\") " pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp"
Oct 30 13:35:57.494945 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:57.494874    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8285d6ec-8ad7-4a33-a696-d32242da2f80-istio-envoy\") pod \"e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp\" (UID: \"8285d6ec-8ad7-4a33-a696-d32242da2f80\") " pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp"
Oct 30 13:35:57.494945 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:57.494898    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8285d6ec-8ad7-4a33-a696-d32242da2f80-workload-certs\") pod \"e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp\" (UID: \"8285d6ec-8ad7-4a33-a696-d32242da2f80\") " pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp"
Oct 30 13:35:57.495234 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:57.494952    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8285d6ec-8ad7-4a33-a696-d32242da2f80-istio-token\") pod \"e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp\" (UID: \"8285d6ec-8ad7-4a33-a696-d32242da2f80\") " pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp"
Oct 30 13:35:57.596747 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:57.596691    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8285d6ec-8ad7-4a33-a696-d32242da2f80-credential-socket\") pod \"e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp\" (UID: \"8285d6ec-8ad7-4a33-a696-d32242da2f80\") " pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp"
Oct 30 13:35:57.596961 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:57.596855    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8285d6ec-8ad7-4a33-a696-d32242da2f80-workload-socket\") pod \"e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp\" (UID: \"8285d6ec-8ad7-4a33-a696-d32242da2f80\") " pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp"
Oct 30 13:35:57.596961 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:57.596903    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8285d6ec-8ad7-4a33-a696-d32242da2f80-istio-envoy\") pod \"e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp\" (UID: \"8285d6ec-8ad7-4a33-a696-d32242da2f80\") " pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp"
Oct 30 13:35:57.596961 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:57.596930    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8285d6ec-8ad7-4a33-a696-d32242da2f80-workload-certs\") pod \"e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp\" (UID: \"8285d6ec-8ad7-4a33-a696-d32242da2f80\") " pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp"
Oct 30 13:35:57.596961 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:57.596959    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8285d6ec-8ad7-4a33-a696-d32242da2f80-istio-token\") pod \"e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp\" (UID: \"8285d6ec-8ad7-4a33-a696-d32242da2f80\") " pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp"
Oct 30 13:35:57.597206 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:57.597170    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8285d6ec-8ad7-4a33-a696-d32242da2f80-istiod-ca-cert\") pod \"e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp\" (UID: \"8285d6ec-8ad7-4a33-a696-d32242da2f80\") " pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp"
Oct 30 13:35:57.597206 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:57.597181    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8285d6ec-8ad7-4a33-a696-d32242da2f80-credential-socket\") pod \"e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp\" (UID: \"8285d6ec-8ad7-4a33-a696-d32242da2f80\") " pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp"
Oct 30 13:35:57.597332 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:57.597206    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8285d6ec-8ad7-4a33-a696-d32242da2f80-istio-data\") pod \"e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp\" (UID: \"8285d6ec-8ad7-4a33-a696-d32242da2f80\") " pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp"
Oct 30 13:35:57.597332 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:57.597259    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8285d6ec-8ad7-4a33-a696-d32242da2f80-workload-socket\") pod \"e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp\" (UID: \"8285d6ec-8ad7-4a33-a696-d32242da2f80\") " pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp"
Oct 30 13:35:57.597332 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:57.597302    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c8pm\" (UniqueName: \"kubernetes.io/projected/8285d6ec-8ad7-4a33-a696-d32242da2f80-kube-api-access-2c8pm\") pod \"e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp\" (UID: \"8285d6ec-8ad7-4a33-a696-d32242da2f80\") " pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp"
Oct 30 13:35:57.597332 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:57.597340    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8285d6ec-8ad7-4a33-a696-d32242da2f80-istio-podinfo\") pod \"e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp\" (UID: \"8285d6ec-8ad7-4a33-a696-d32242da2f80\") " pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp"
Oct 30 13:35:57.597550 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:57.597368    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8285d6ec-8ad7-4a33-a696-d32242da2f80-workload-certs\") pod \"e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp\" (UID: \"8285d6ec-8ad7-4a33-a696-d32242da2f80\") " pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp"
Oct 30 13:35:57.597604 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:57.597545    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8285d6ec-8ad7-4a33-a696-d32242da2f80-istio-data\") pod \"e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp\" (UID: \"8285d6ec-8ad7-4a33-a696-d32242da2f80\") " pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp"
Oct 30 13:35:57.598277 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:57.598248    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8285d6ec-8ad7-4a33-a696-d32242da2f80-istiod-ca-cert\") pod \"e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp\" (UID: \"8285d6ec-8ad7-4a33-a696-d32242da2f80\") " pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp"
Oct 30 13:35:57.599919 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:57.599892    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8285d6ec-8ad7-4a33-a696-d32242da2f80-istio-envoy\") pod \"e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp\" (UID: \"8285d6ec-8ad7-4a33-a696-d32242da2f80\") " pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp"
Oct 30 13:35:57.600025 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:57.599986    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8285d6ec-8ad7-4a33-a696-d32242da2f80-istio-podinfo\") pod \"e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp\" (UID: \"8285d6ec-8ad7-4a33-a696-d32242da2f80\") " pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp"
Oct 30 13:35:57.618196 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:57.618167    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8285d6ec-8ad7-4a33-a696-d32242da2f80-istio-token\") pod \"e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp\" (UID: \"8285d6ec-8ad7-4a33-a696-d32242da2f80\") " pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp"
Oct 30 13:35:57.618334 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:57.618266    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c8pm\" (UniqueName: \"kubernetes.io/projected/8285d6ec-8ad7-4a33-a696-d32242da2f80-kube-api-access-2c8pm\") pod \"e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp\" (UID: \"8285d6ec-8ad7-4a33-a696-d32242da2f80\") " pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp"
Oct 30 13:35:57.781464 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:57.781385    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp"
Oct 30 13:35:57.924237 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:57.924208    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-bd2fed-68ff7bd4b6-d9tt2"
Oct 30 13:35:58.106244 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:58.106212    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp"]
Oct 30 13:35:58.113630 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:58.113585    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:35:58.113768 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:58.113727    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:35:58.113836 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:58.113818    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:35:59.035792 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:59.035753    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp" event={"ID":"8285d6ec-8ad7-4a33-a696-d32242da2f80","Type":"ContainerStarted","Data":"58cb4ccd0578760096ff3f24e0e118b4a3cbe5cdc22db71b7f3feb99caa1207b"}
Oct 30 13:35:59.036229 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:59.035813    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp" event={"ID":"8285d6ec-8ad7-4a33-a696-d32242da2f80","Type":"ContainerStarted","Data":"018d07c6acc575d4a0da1e956ce6dc7905628931f121fba03a9dbd8ea727a83c"}
Oct 30 13:35:59.096806 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:59.096739    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp" podStartSLOduration=2.096719186 podStartE2EDuration="2.096719186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:35:59.094550323 +0000 UTC m=+1978.241817558" watchObservedRunningTime="2026-10-30 13:35:59.096719186 +0000 UTC m=+1978.243986428"
Oct 30 13:35:59.782530 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:59.782494    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp"
Oct 30 13:35:59.789835 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:35:59.789807    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp"
Oct 30 13:36:00.044377 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:00.044293    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp"
Oct 30 13:36:00.045497 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:00.045470    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp"
Oct 30 13:36:00.960926 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:00.960895    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-310546-7c9f444fc9-xw4r4"
Oct 30 13:36:01.977375 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:01.977343    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-310546-6db5dd49d6-xfhp4"
Oct 30 13:36:12.057216 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:12.057169    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd2fed-67b8dc8f65-v5kx7"]
Oct 30 13:36:12.069985 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:12.069931    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd2fed-67b8dc8f65-v5kx7"
Oct 30 13:36:12.077385 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:12.077354    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd2fed-67b8dc8f65-v5kx7"]
Oct 30 13:36:12.090821 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:12.089689    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-bd2fed-serving-cert\""
Oct 30 13:36:12.162913 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:12.162792    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00f2b03f-864c-4e03-9144-b949bced2ba9-metrics-certs\") pod \"maas-api-e2e-extmod-a-bd2fed-67b8dc8f65-v5kx7\" (UID: \"00f2b03f-864c-4e03-9144-b949bced2ba9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd2fed-67b8dc8f65-v5kx7"
Oct 30 13:36:12.162913 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:12.162894    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00f2b03f-864c-4e03-9144-b949bced2ba9-maas-api-tls\") pod \"maas-api-e2e-extmod-a-bd2fed-67b8dc8f65-v5kx7\" (UID: \"00f2b03f-864c-4e03-9144-b949bced2ba9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd2fed-67b8dc8f65-v5kx7"
Oct 30 13:36:12.163344 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:12.162956    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmtbf\" (UniqueName: \"kubernetes.io/projected/00f2b03f-864c-4e03-9144-b949bced2ba9-kube-api-access-vmtbf\") pod \"maas-api-e2e-extmod-a-bd2fed-67b8dc8f65-v5kx7\" (UID: \"00f2b03f-864c-4e03-9144-b949bced2ba9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd2fed-67b8dc8f65-v5kx7"
Oct 30 13:36:12.163344 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:12.163028    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00f2b03f-864c-4e03-9144-b949bced2ba9-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-bd2fed-67b8dc8f65-v5kx7\" (UID: \"00f2b03f-864c-4e03-9144-b949bced2ba9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd2fed-67b8dc8f65-v5kx7"
Oct 30 13:36:12.264834 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:12.264792    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmtbf\" (UniqueName: \"kubernetes.io/projected/00f2b03f-864c-4e03-9144-b949bced2ba9-kube-api-access-vmtbf\") pod \"maas-api-e2e-extmod-a-bd2fed-67b8dc8f65-v5kx7\" (UID: \"00f2b03f-864c-4e03-9144-b949bced2ba9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd2fed-67b8dc8f65-v5kx7"
Oct 30 13:36:12.265212 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:12.264946    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00f2b03f-864c-4e03-9144-b949bced2ba9-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-bd2fed-67b8dc8f65-v5kx7\" (UID: \"00f2b03f-864c-4e03-9144-b949bced2ba9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd2fed-67b8dc8f65-v5kx7"
Oct 30 13:36:12.265212 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:12.265102    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00f2b03f-864c-4e03-9144-b949bced2ba9-metrics-certs\") pod \"maas-api-e2e-extmod-a-bd2fed-67b8dc8f65-v5kx7\" (UID: \"00f2b03f-864c-4e03-9144-b949bced2ba9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd2fed-67b8dc8f65-v5kx7"
Oct 30 13:36:12.265376 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:12.265220    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00f2b03f-864c-4e03-9144-b949bced2ba9-maas-api-tls\") pod \"maas-api-e2e-extmod-a-bd2fed-67b8dc8f65-v5kx7\" (UID: \"00f2b03f-864c-4e03-9144-b949bced2ba9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd2fed-67b8dc8f65-v5kx7"
Oct 30 13:36:12.286533 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:12.285729    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00f2b03f-864c-4e03-9144-b949bced2ba9-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-bd2fed-67b8dc8f65-v5kx7\" (UID: \"00f2b03f-864c-4e03-9144-b949bced2ba9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd2fed-67b8dc8f65-v5kx7"
Oct 30 13:36:12.290164 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:12.288110    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00f2b03f-864c-4e03-9144-b949bced2ba9-maas-api-tls\") pod \"maas-api-e2e-extmod-a-bd2fed-67b8dc8f65-v5kx7\" (UID: \"00f2b03f-864c-4e03-9144-b949bced2ba9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd2fed-67b8dc8f65-v5kx7"
Oct 30 13:36:12.290164 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:12.288981    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00f2b03f-864c-4e03-9144-b949bced2ba9-metrics-certs\") pod \"maas-api-e2e-extmod-a-bd2fed-67b8dc8f65-v5kx7\" (UID: \"00f2b03f-864c-4e03-9144-b949bced2ba9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd2fed-67b8dc8f65-v5kx7"
Oct 30 13:36:12.306973 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:12.306940    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmtbf\" (UniqueName: \"kubernetes.io/projected/00f2b03f-864c-4e03-9144-b949bced2ba9-kube-api-access-vmtbf\") pod \"maas-api-e2e-extmod-a-bd2fed-67b8dc8f65-v5kx7\" (UID: \"00f2b03f-864c-4e03-9144-b949bced2ba9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd2fed-67b8dc8f65-v5kx7"
Oct 30 13:36:12.420394 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:12.420296    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd2fed-67b8dc8f65-v5kx7"
Oct 30 13:36:12.778163 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:12.775700    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd2fed-67b8dc8f65-v5kx7"]
Oct 30 13:36:12.781865 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:36:12.781825    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00f2b03f_864c_4e03_9144_b949bced2ba9.slice/crio-ef56e186ea5b1d1612d556b48f438904472c7e58ee93cb0605e232308f186188 WatchSource:0}: Error finding container ef56e186ea5b1d1612d556b48f438904472c7e58ee93cb0605e232308f186188: Status 404 returned error can't find the container with id ef56e186ea5b1d1612d556b48f438904472c7e58ee93cb0605e232308f186188
Oct 30 13:36:12.783797 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:12.783772    5093 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 13:36:13.191155 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:13.185443    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd2fed-67b8dc8f65-v5kx7" event={"ID":"00f2b03f-864c-4e03-9144-b949bced2ba9","Type":"ContainerStarted","Data":"ef56e186ea5b1d1612d556b48f438904472c7e58ee93cb0605e232308f186188"}
Oct 30 13:36:14.194311 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:14.194269    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd2fed-67b8dc8f65-v5kx7" event={"ID":"00f2b03f-864c-4e03-9144-b949bced2ba9","Type":"ContainerStarted","Data":"673eee8ca1cbb0b5404386eb25583c30c116052ec63261f8991aac7296a4f7f0"}
Oct 30 13:36:14.194778 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:14.194364    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd2fed-67b8dc8f65-v5kx7"
Oct 30 13:36:14.236584 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:14.236528    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd2fed-67b8dc8f65-v5kx7" podStartSLOduration=1.829231308 podStartE2EDuration="2.236513538s" totalImagesPullingTime="407.28223ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:36:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:36:14.234864153 +0000 UTC m=+1993.382131397" watchObservedRunningTime="2026-10-30 13:36:14.236513538 +0000 UTC m=+1993.383780778"
Oct 30 13:36:16.332973 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:16.332931    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-310546-7f57cb9f95-5ghvp"]
Oct 30 13:36:16.338375 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:16.338348    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-310546-7f57cb9f95-5ghvp"
Oct 30 13:36:16.370917 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:16.370864    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-310546-7f57cb9f95-5ghvp"]
Oct 30 13:36:16.371551 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:16.371512    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-310546-serving-cert\""
Oct 30 13:36:16.482454 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:16.482418    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b819faba-b9f4-488f-bfa0-4da40fb59313-metrics-certs\") pod \"maas-api-e2e-extmod-a-310546-7f57cb9f95-5ghvp\" (UID: \"b819faba-b9f4-488f-bfa0-4da40fb59313\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-310546-7f57cb9f95-5ghvp"
Oct 30 13:36:16.482641 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:16.482556    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b819faba-b9f4-488f-bfa0-4da40fb59313-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-310546-7f57cb9f95-5ghvp\" (UID: \"b819faba-b9f4-488f-bfa0-4da40fb59313\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-310546-7f57cb9f95-5ghvp"
Oct 30 13:36:16.482685 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:16.482665    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcsqw\" (UniqueName: \"kubernetes.io/projected/b819faba-b9f4-488f-bfa0-4da40fb59313-kube-api-access-pcsqw\") pod \"maas-api-e2e-extmod-a-310546-7f57cb9f95-5ghvp\" (UID: \"b819faba-b9f4-488f-bfa0-4da40fb59313\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-310546-7f57cb9f95-5ghvp"
Oct 30 13:36:16.482772 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:16.482757    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b819faba-b9f4-488f-bfa0-4da40fb59313-maas-api-tls\") pod \"maas-api-e2e-extmod-a-310546-7f57cb9f95-5ghvp\" (UID: \"b819faba-b9f4-488f-bfa0-4da40fb59313\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-310546-7f57cb9f95-5ghvp"
Oct 30 13:36:16.584613 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:16.584515    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b819faba-b9f4-488f-bfa0-4da40fb59313-metrics-certs\") pod \"maas-api-e2e-extmod-a-310546-7f57cb9f95-5ghvp\" (UID: \"b819faba-b9f4-488f-bfa0-4da40fb59313\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-310546-7f57cb9f95-5ghvp"
Oct 30 13:36:16.584803 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:16.584633    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b819faba-b9f4-488f-bfa0-4da40fb59313-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-310546-7f57cb9f95-5ghvp\" (UID: \"b819faba-b9f4-488f-bfa0-4da40fb59313\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-310546-7f57cb9f95-5ghvp"
Oct 30 13:36:16.584803 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:16.584731    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcsqw\" (UniqueName: \"kubernetes.io/projected/b819faba-b9f4-488f-bfa0-4da40fb59313-kube-api-access-pcsqw\") pod \"maas-api-e2e-extmod-a-310546-7f57cb9f95-5ghvp\" (UID: \"b819faba-b9f4-488f-bfa0-4da40fb59313\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-310546-7f57cb9f95-5ghvp"
Oct 30 13:36:16.584990 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:16.584845    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b819faba-b9f4-488f-bfa0-4da40fb59313-maas-api-tls\") pod \"maas-api-e2e-extmod-a-310546-7f57cb9f95-5ghvp\" (UID: \"b819faba-b9f4-488f-bfa0-4da40fb59313\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-310546-7f57cb9f95-5ghvp"
Oct 30 13:36:16.585326 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:16.585299    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b819faba-b9f4-488f-bfa0-4da40fb59313-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-310546-7f57cb9f95-5ghvp\" (UID: \"b819faba-b9f4-488f-bfa0-4da40fb59313\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-310546-7f57cb9f95-5ghvp"
Oct 30 13:36:16.587142 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:16.587102    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b819faba-b9f4-488f-bfa0-4da40fb59313-metrics-certs\") pod \"maas-api-e2e-extmod-a-310546-7f57cb9f95-5ghvp\" (UID: \"b819faba-b9f4-488f-bfa0-4da40fb59313\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-310546-7f57cb9f95-5ghvp"
Oct 30 13:36:16.587246 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:16.587180    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b819faba-b9f4-488f-bfa0-4da40fb59313-maas-api-tls\") pod \"maas-api-e2e-extmod-a-310546-7f57cb9f95-5ghvp\" (UID: \"b819faba-b9f4-488f-bfa0-4da40fb59313\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-310546-7f57cb9f95-5ghvp"
Oct 30 13:36:16.622785 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:16.622712    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcsqw\" (UniqueName: \"kubernetes.io/projected/b819faba-b9f4-488f-bfa0-4da40fb59313-kube-api-access-pcsqw\") pod \"maas-api-e2e-extmod-a-310546-7f57cb9f95-5ghvp\" (UID: \"b819faba-b9f4-488f-bfa0-4da40fb59313\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-310546-7f57cb9f95-5ghvp"
Oct 30 13:36:16.676799 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:16.676754    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-310546-7f57cb9f95-5ghvp"
Oct 30 13:36:17.000098 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:36:17.000067    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb819faba_b9f4_488f_bfa0_4da40fb59313.slice/crio-8505c865ae1e0cd5aedda4e7b8ebc09c54d73f58889938b3585b8118549d8b67 WatchSource:0}: Error finding container 8505c865ae1e0cd5aedda4e7b8ebc09c54d73f58889938b3585b8118549d8b67: Status 404 returned error can't find the container with id 8505c865ae1e0cd5aedda4e7b8ebc09c54d73f58889938b3585b8118549d8b67
Oct 30 13:36:17.005067 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:17.005043    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-310546-7f57cb9f95-5ghvp"]
Oct 30 13:36:17.220270 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:17.220232    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-310546-7f57cb9f95-5ghvp" event={"ID":"b819faba-b9f4-488f-bfa0-4da40fb59313","Type":"ContainerStarted","Data":"8505c865ae1e0cd5aedda4e7b8ebc09c54d73f58889938b3585b8118549d8b67"}
Oct 30 13:36:18.230897 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:18.230860    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-310546-7f57cb9f95-5ghvp" event={"ID":"b819faba-b9f4-488f-bfa0-4da40fb59313","Type":"ContainerStarted","Data":"58a1c5009b3015c501c0e183d602e2576f1bc53570e7616fbb66fcd1c8838838"}
Oct 30 13:36:18.231381 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:18.230941    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-310546-7f57cb9f95-5ghvp"
Oct 30 13:36:18.279426 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:18.279343    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-310546-7f57cb9f95-5ghvp" podStartSLOduration=1.9680487530000002 podStartE2EDuration="2.279321349s" totalImagesPullingTime="311.272596ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:36:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:36:18.275700864 +0000 UTC m=+1997.422968106" watchObservedRunningTime="2026-10-30 13:36:18.279321349 +0000 UTC m=+1997.426588592"
Oct 30 13:36:20.208606 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:20.208576    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd2fed-67b8dc8f65-v5kx7"
Oct 30 13:36:21.172310 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:36:21.172269    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:36:24.250186 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:24.250123    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-310546-7f57cb9f95-5ghvp"
Oct 30 13:36:31.724451 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:31.724415    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn"]
Oct 30 13:36:31.724884 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:31.724635    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn" podUID="d087f5d8-cdf1-4014-98ab-8aa0170489c9" containerName="istio-proxy" containerID="cri-o://199faf04f21057ea20fb54422b914fb39c18061608a7b29b189a31e311d460f8" gracePeriod=30
Oct 30 13:36:37.035563 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:37.035529    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn"
Oct 30 13:36:37.137460 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:37.137417    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d087f5d8-cdf1-4014-98ab-8aa0170489c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d087f5d8-cdf1-4014-98ab-8aa0170489c9-workload-socket\") pod \"d087f5d8-cdf1-4014-98ab-8aa0170489c9\" (UID: \"d087f5d8-cdf1-4014-98ab-8aa0170489c9\") "
Oct 30 13:36:37.137759 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:37.137719    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d087f5d8-cdf1-4014-98ab-8aa0170489c9-workload-socket" pod "d087f5d8-cdf1-4014-98ab-8aa0170489c9" (UID: "d087f5d8-cdf1-4014-98ab-8aa0170489c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:37.137759 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:37.137746    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d087f5d8-cdf1-4014-98ab-8aa0170489c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d087f5d8-cdf1-4014-98ab-8aa0170489c9-istio-data\") pod \"d087f5d8-cdf1-4014-98ab-8aa0170489c9\" (UID: \"d087f5d8-cdf1-4014-98ab-8aa0170489c9\") "
Oct 30 13:36:37.137942 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:37.137800    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d087f5d8-cdf1-4014-98ab-8aa0170489c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d087f5d8-cdf1-4014-98ab-8aa0170489c9-workload-certs\") pod \"d087f5d8-cdf1-4014-98ab-8aa0170489c9\" (UID: \"d087f5d8-cdf1-4014-98ab-8aa0170489c9\") "
Oct 30 13:36:37.137942 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:37.137874    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d087f5d8-cdf1-4014-98ab-8aa0170489c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d087f5d8-cdf1-4014-98ab-8aa0170489c9-istiod-ca-cert\") pod \"d087f5d8-cdf1-4014-98ab-8aa0170489c9\" (UID: \"d087f5d8-cdf1-4014-98ab-8aa0170489c9\") "
Oct 30 13:36:37.137942 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:37.137920    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d087f5d8-cdf1-4014-98ab-8aa0170489c9-istio-data" pod "d087f5d8-cdf1-4014-98ab-8aa0170489c9" (UID: "d087f5d8-cdf1-4014-98ab-8aa0170489c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:37.138122 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:37.137966    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d087f5d8-cdf1-4014-98ab-8aa0170489c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d087f5d8-cdf1-4014-98ab-8aa0170489c9-istio-envoy\") pod \"d087f5d8-cdf1-4014-98ab-8aa0170489c9\" (UID: \"d087f5d8-cdf1-4014-98ab-8aa0170489c9\") "
Oct 30 13:36:37.138122 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:37.138028    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d087f5d8-cdf1-4014-98ab-8aa0170489c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d087f5d8-cdf1-4014-98ab-8aa0170489c9-credential-socket\") pod \"d087f5d8-cdf1-4014-98ab-8aa0170489c9\" (UID: \"d087f5d8-cdf1-4014-98ab-8aa0170489c9\") "
Oct 30 13:36:37.138279 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:37.138147    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d087f5d8-cdf1-4014-98ab-8aa0170489c9-workload-certs" pod "d087f5d8-cdf1-4014-98ab-8aa0170489c9" (UID: "d087f5d8-cdf1-4014-98ab-8aa0170489c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:37.138279 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:37.138161    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d087f5d8-cdf1-4014-98ab-8aa0170489c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d087f5d8-cdf1-4014-98ab-8aa0170489c9-istio-podinfo\") pod \"d087f5d8-cdf1-4014-98ab-8aa0170489c9\" (UID: \"d087f5d8-cdf1-4014-98ab-8aa0170489c9\") "
Oct 30 13:36:37.138279 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:37.138233    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d087f5d8-cdf1-4014-98ab-8aa0170489c9-kube-api-access-pgvdr\" (UniqueName: \"kubernetes.io/projected/d087f5d8-cdf1-4014-98ab-8aa0170489c9-kube-api-access-pgvdr\") pod \"d087f5d8-cdf1-4014-98ab-8aa0170489c9\" (UID: \"d087f5d8-cdf1-4014-98ab-8aa0170489c9\") "
Oct 30 13:36:37.138436 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:37.138279    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d087f5d8-cdf1-4014-98ab-8aa0170489c9-istio-token\" (UniqueName: \"kubernetes.io/projected/d087f5d8-cdf1-4014-98ab-8aa0170489c9-istio-token\") pod \"d087f5d8-cdf1-4014-98ab-8aa0170489c9\" (UID: \"d087f5d8-cdf1-4014-98ab-8aa0170489c9\") "
Oct 30 13:36:37.139107 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:37.138729    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d087f5d8-cdf1-4014-98ab-8aa0170489c9-istiod-ca-cert" pod "d087f5d8-cdf1-4014-98ab-8aa0170489c9" (UID: "d087f5d8-cdf1-4014-98ab-8aa0170489c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:36:37.139107 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:37.138946    5093 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d087f5d8-cdf1-4014-98ab-8aa0170489c9-workload-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:36:37.139107 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:37.138964    5093 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d087f5d8-cdf1-4014-98ab-8aa0170489c9-istio-data\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:36:37.139107 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:37.138978    5093 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d087f5d8-cdf1-4014-98ab-8aa0170489c9-workload-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:36:37.139107 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:37.139092    5093 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d087f5d8-cdf1-4014-98ab-8aa0170489c9-istiod-ca-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:36:37.139432 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:37.139086    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d087f5d8-cdf1-4014-98ab-8aa0170489c9-credential-socket" pod "d087f5d8-cdf1-4014-98ab-8aa0170489c9" (UID: "d087f5d8-cdf1-4014-98ab-8aa0170489c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:37.141702 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:37.141667    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d087f5d8-cdf1-4014-98ab-8aa0170489c9-istio-token" pod "d087f5d8-cdf1-4014-98ab-8aa0170489c9" (UID: "d087f5d8-cdf1-4014-98ab-8aa0170489c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:36:37.142962 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:37.142200    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d087f5d8-cdf1-4014-98ab-8aa0170489c9-istio-envoy" pod "d087f5d8-cdf1-4014-98ab-8aa0170489c9" (UID: "d087f5d8-cdf1-4014-98ab-8aa0170489c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:37.142962 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:37.142531    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d087f5d8-cdf1-4014-98ab-8aa0170489c9-kube-api-access-pgvdr" pod "d087f5d8-cdf1-4014-98ab-8aa0170489c9" (UID: "d087f5d8-cdf1-4014-98ab-8aa0170489c9"). InnerVolumeSpecName "kube-api-access-pgvdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:36:37.144883 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:37.144850    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d087f5d8-cdf1-4014-98ab-8aa0170489c9-istio-podinfo" pod "d087f5d8-cdf1-4014-98ab-8aa0170489c9" (UID: "d087f5d8-cdf1-4014-98ab-8aa0170489c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:36:37.240499 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:37.240466    5093 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d087f5d8-cdf1-4014-98ab-8aa0170489c9-istio-envoy\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:36:37.240499 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:37.240498    5093 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d087f5d8-cdf1-4014-98ab-8aa0170489c9-credential-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:36:37.240710 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:37.240517    5093 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d087f5d8-cdf1-4014-98ab-8aa0170489c9-istio-podinfo\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:36:37.240710 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:37.240528    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgvdr\" (UniqueName: \"kubernetes.io/projected/d087f5d8-cdf1-4014-98ab-8aa0170489c9-kube-api-access-pgvdr\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:36:37.240710 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:37.240538    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d087f5d8-cdf1-4014-98ab-8aa0170489c9-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:36:37.399438 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:37.399348    5093 generic.go:356] "Generic (PLEG): container finished" podID="d087f5d8-cdf1-4014-98ab-8aa0170489c9" containerID="199faf04f21057ea20fb54422b914fb39c18061608a7b29b189a31e311d460f8" exitCode=0
Oct 30 13:36:37.399438 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:37.399402    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn"
Oct 30 13:36:37.399626 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:37.399437    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn" event={"ID":"d087f5d8-cdf1-4014-98ab-8aa0170489c9","Type":"ContainerDied","Data":"199faf04f21057ea20fb54422b914fb39c18061608a7b29b189a31e311d460f8"}
Oct 30 13:36:37.399626 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:37.399494    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn" event={"ID":"d087f5d8-cdf1-4014-98ab-8aa0170489c9","Type":"ContainerDied","Data":"c195c243fbe29330898a8c733183a3379976154b42201867ae01aa2ccedd57e6"}
Oct 30 13:36:37.399626 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:37.399522    5093 scope.go:122] "RemoveContainer" containerID="199faf04f21057ea20fb54422b914fb39c18061608a7b29b189a31e311d460f8"
Oct 30 13:36:37.413625 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:37.413600    5093 scope.go:122] "RemoveContainer" containerID="199faf04f21057ea20fb54422b914fb39c18061608a7b29b189a31e311d460f8"
Oct 30 13:36:37.413911 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:36:37.413891    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"199faf04f21057ea20fb54422b914fb39c18061608a7b29b189a31e311d460f8\": container with ID starting with 199faf04f21057ea20fb54422b914fb39c18061608a7b29b189a31e311d460f8 not found: ID does not exist" containerID="199faf04f21057ea20fb54422b914fb39c18061608a7b29b189a31e311d460f8"
Oct 30 13:36:37.413988 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:37.413919    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"199faf04f21057ea20fb54422b914fb39c18061608a7b29b189a31e311d460f8"} err="failed to get container status \"199faf04f21057ea20fb54422b914fb39c18061608a7b29b189a31e311d460f8\": rpc error: code = NotFound desc = could not find container \"199faf04f21057ea20fb54422b914fb39c18061608a7b29b189a31e311d460f8\": container with ID starting with 199faf04f21057ea20fb54422b914fb39c18061608a7b29b189a31e311d460f8 not found: ID does not exist"
Oct 30 13:36:37.456621 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:37.456576    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn"]
Oct 30 13:36:37.460619 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:37.460587    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-j6txn"]
Oct 30 13:36:39.003772 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:39.003740    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d087f5d8-cdf1-4014-98ab-8aa0170489c9" path="/var/lib/kubelet/pods/d087f5d8-cdf1-4014-98ab-8aa0170489c9/volumes"
Oct 30 13:36:42.089108 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:42.089025    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-f7ee8a/auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz"]
Oct 30 13:36:51.171424 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:36:51.171378    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:36:53.540455 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:53.540366    5093 generic.go:356] "Generic (PLEG): container finished" podID="18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e" containerID="bf678e91aa9af56a31ab09de6b7287b69bc50de05e9cf7030530a2bf694faf47" exitCode=0
Oct 30 13:36:53.540922 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:53.540439    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp" event={"ID":"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e","Type":"ContainerDied","Data":"bf678e91aa9af56a31ab09de6b7287b69bc50de05e9cf7030530a2bf694faf47"}
Oct 30 13:36:53.540998 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:53.540953    5093 scope.go:122] "RemoveContainer" containerID="bf678e91aa9af56a31ab09de6b7287b69bc50de05e9cf7030530a2bf694faf47"
Oct 30 13:36:53.542912 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:53.542880    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:36:53.543042 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:53.542972    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:36:53.543042 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:53.543022    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:36:54.624159 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:54.622545    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp" event={"ID":"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e","Type":"ContainerStarted","Data":"3ff07d3b7abb82a8b1e62ed1fadc16ab071c289b98414c744a1ad42ea3357287"}
Oct 30 13:36:55.473207 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:55.473164    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp"
Oct 30 13:36:55.474818 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:55.474788    5093 patch_prober.go:29] interesting pod/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 13:36:55.474961 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:55.474839    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp" podUID="18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 13:36:56.473745 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:56.473707    5093 patch_prober.go:29] interesting pod/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 13:36:56.474312 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:56.473774    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp" podUID="18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 13:36:56.852271 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:56.852089    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp"
Oct 30 13:36:57.118230 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:57.118108    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-f7ee8a/auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz" podUID="ed392ef1-7739-4131-9e4b-9568d9e68801" containerName="main" containerID="cri-o://50e24476000696345ec9607a66e83c774dfae3403d41ba7f6d38d0cb96eecbfa" gracePeriod=45
Oct 30 13:36:57.473743 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:57.473700    5093 patch_prober.go:29] interesting pod/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 13:36:57.473916 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:57.473772    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp" podUID="18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 13:36:57.683120 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:57.683043    5093 generic.go:356] "Generic (PLEG): container finished" podID="ed392ef1-7739-4131-9e4b-9568d9e68801" containerID="50e24476000696345ec9607a66e83c774dfae3403d41ba7f6d38d0cb96eecbfa" exitCode=0
Oct 30 13:36:57.683298 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:57.683115    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-f7ee8a/auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz" event={"ID":"ed392ef1-7739-4131-9e4b-9568d9e68801","Type":"ContainerDied","Data":"50e24476000696345ec9607a66e83c774dfae3403d41ba7f6d38d0cb96eecbfa"}
Oct 30 13:36:57.683298 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:57.683183    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-f7ee8a/auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz" event={"ID":"ed392ef1-7739-4131-9e4b-9568d9e68801","Type":"ContainerDied","Data":"56c35a372e7422e742f6e20270e6a93264b503493df03d9f2b1daea1b5cb6de5"}
Oct 30 13:36:57.683298 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:57.683199    5093 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56c35a372e7422e742f6e20270e6a93264b503493df03d9f2b1daea1b5cb6de5"
Oct 30 13:36:57.691539 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:57.691515    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-f7ee8a/auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz"
Oct 30 13:36:57.765805 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:57.765728    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed392ef1-7739-4131-9e4b-9568d9e68801-home\" (UniqueName: \"kubernetes.io/empty-dir/ed392ef1-7739-4131-9e4b-9568d9e68801-home\") pod \"ed392ef1-7739-4131-9e4b-9568d9e68801\" (UID: \"ed392ef1-7739-4131-9e4b-9568d9e68801\") "
Oct 30 13:36:57.765998 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:57.765847    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed392ef1-7739-4131-9e4b-9568d9e68801-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed392ef1-7739-4131-9e4b-9568d9e68801-dshm\") pod \"ed392ef1-7739-4131-9e4b-9568d9e68801\" (UID: \"ed392ef1-7739-4131-9e4b-9568d9e68801\") "
Oct 30 13:36:57.765998 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:57.765963    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed392ef1-7739-4131-9e4b-9568d9e68801-tls-certs\" (UniqueName: \"kubernetes.io/secret/ed392ef1-7739-4131-9e4b-9568d9e68801-tls-certs\") pod \"ed392ef1-7739-4131-9e4b-9568d9e68801\" (UID: \"ed392ef1-7739-4131-9e4b-9568d9e68801\") "
Oct 30 13:36:57.766176 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:57.766106    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed392ef1-7739-4131-9e4b-9568d9e68801-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed392ef1-7739-4131-9e4b-9568d9e68801-model-cache\") pod \"ed392ef1-7739-4131-9e4b-9568d9e68801\" (UID: \"ed392ef1-7739-4131-9e4b-9568d9e68801\") "
Oct 30 13:36:57.766266 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:57.766246    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed392ef1-7739-4131-9e4b-9568d9e68801-kube-api-access-bt5tf\" (UniqueName: \"kubernetes.io/projected/ed392ef1-7739-4131-9e4b-9568d9e68801-kube-api-access-bt5tf\") pod \"ed392ef1-7739-4131-9e4b-9568d9e68801\" (UID: \"ed392ef1-7739-4131-9e4b-9568d9e68801\") "
Oct 30 13:36:57.766336 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:57.766311    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed392ef1-7739-4131-9e4b-9568d9e68801-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed392ef1-7739-4131-9e4b-9568d9e68801-tmp-dir\") pod \"ed392ef1-7739-4131-9e4b-9568d9e68801\" (UID: \"ed392ef1-7739-4131-9e4b-9568d9e68801\") "
Oct 30 13:36:57.766577 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:57.766548    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed392ef1-7739-4131-9e4b-9568d9e68801-home" pod "ed392ef1-7739-4131-9e4b-9568d9e68801" (UID: "ed392ef1-7739-4131-9e4b-9568d9e68801"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:57.766871 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:57.766836    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed392ef1-7739-4131-9e4b-9568d9e68801-model-cache" pod "ed392ef1-7739-4131-9e4b-9568d9e68801" (UID: "ed392ef1-7739-4131-9e4b-9568d9e68801"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:57.767032 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:57.767002    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed392ef1-7739-4131-9e4b-9568d9e68801-tmp-dir" pod "ed392ef1-7739-4131-9e4b-9568d9e68801" (UID: "ed392ef1-7739-4131-9e4b-9568d9e68801"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:57.767172 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:57.767146    5093 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed392ef1-7739-4131-9e4b-9568d9e68801-home\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:36:57.767249 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:57.767171    5093 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed392ef1-7739-4131-9e4b-9568d9e68801-model-cache\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:36:57.767249 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:57.767186    5093 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed392ef1-7739-4131-9e4b-9568d9e68801-tmp-dir\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:36:57.769949 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:57.769921    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed392ef1-7739-4131-9e4b-9568d9e68801-tls-certs" pod "ed392ef1-7739-4131-9e4b-9568d9e68801" (UID: "ed392ef1-7739-4131-9e4b-9568d9e68801"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:36:57.770744 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:57.770716    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed392ef1-7739-4131-9e4b-9568d9e68801-kube-api-access-bt5tf" pod "ed392ef1-7739-4131-9e4b-9568d9e68801" (UID: "ed392ef1-7739-4131-9e4b-9568d9e68801"). InnerVolumeSpecName "kube-api-access-bt5tf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:36:57.772376 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:57.772344    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed392ef1-7739-4131-9e4b-9568d9e68801-dshm" pod "ed392ef1-7739-4131-9e4b-9568d9e68801" (UID: "ed392ef1-7739-4131-9e4b-9568d9e68801"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:57.868428 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:57.868389    5093 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed392ef1-7739-4131-9e4b-9568d9e68801-dshm\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:36:57.868428 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:57.868428    5093 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed392ef1-7739-4131-9e4b-9568d9e68801-tls-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:36:57.868651 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:57.868440    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt5tf\" (UniqueName: \"kubernetes.io/projected/ed392ef1-7739-4131-9e4b-9568d9e68801-kube-api-access-bt5tf\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:36:58.473426 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:58.473390    5093 patch_prober.go:29] interesting pod/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 13:36:58.473627 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:58.473463    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp" podUID="18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 13:36:58.695554 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:58.695505    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-f7ee8a/auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz"
Oct 30 13:36:58.768057 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:58.767961    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-f7ee8a/auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz"]
Oct 30 13:36:58.772998 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:58.772970    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-f7ee8a/auth-test-model-f7ee8a-kserve-6764f69c57-zmrmz"]
Oct 30 13:36:59.013182 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:59.012682    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed392ef1-7739-4131-9e4b-9568d9e68801" path="/var/lib/kubelet/pods/ed392ef1-7739-4131-9e4b-9568d9e68801/volumes"
Oct 30 13:36:59.013182 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:59.013118    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-29a3f3/auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc"]
Oct 30 13:36:59.474266 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:59.473982    5093 patch_prober.go:29] interesting pod/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 13:36:59.474266 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:36:59.474082    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp" podUID="18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 13:37:00.485592 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:00.485551    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp"
Oct 30 13:37:00.487214 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:00.487189    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp"
Oct 30 13:37:01.057589 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:01.057535    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e3eb98-798ccfff95-kpvst"]
Oct 30 13:37:01.059413 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:01.059380    5093 cpu_manager.go:414] "containerMap: removing container" podUID="ed392ef1-7739-4131-9e4b-9568d9e68801" containerName="main"
Oct 30 13:37:01.059619 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:01.059600    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed392ef1-7739-4131-9e4b-9568d9e68801" containerName="main"
Oct 30 13:37:01.059858 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:01.059845    5093 cpu_manager.go:414] "containerMap: removing container" podUID="d087f5d8-cdf1-4014-98ab-8aa0170489c9" containerName="istio-proxy"
Oct 30 13:37:01.059942 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:01.059931    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d087f5d8-cdf1-4014-98ab-8aa0170489c9" containerName="istio-proxy"
Oct 30 13:37:01.060377 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:01.060356    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="d087f5d8-cdf1-4014-98ab-8aa0170489c9" containerName="istio-proxy"
Oct 30 13:37:01.060502 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:01.060491    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed392ef1-7739-4131-9e4b-9568d9e68801" containerName="main"
Oct 30 13:37:01.069385 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:01.069352    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e3eb98-798ccfff95-kpvst"
Oct 30 13:37:01.079666 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:01.079632    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e3eb98-798ccfff95-kpvst"]
Oct 30 13:37:01.137027 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:01.136974    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk997\" (UniqueName: \"kubernetes.io/projected/cd554e69-a448-4d73-885a-d1ffcc724c56-kube-api-access-dk997\") pod \"payload-pre-processing-e2e-extmod-b-e3eb98-798ccfff95-kpvst\" (UID: \"cd554e69-a448-4d73-885a-d1ffcc724c56\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e3eb98-798ccfff95-kpvst"
Oct 30 13:37:01.137267 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:01.137098    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd554e69-a448-4d73-885a-d1ffcc724c56-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-e3eb98-798ccfff95-kpvst\" (UID: \"cd554e69-a448-4d73-885a-d1ffcc724c56\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e3eb98-798ccfff95-kpvst"
Oct 30 13:37:01.169823 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:01.169786    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-e3eb98\""
Oct 30 13:37:01.294156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:01.280602    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk997\" (UniqueName: \"kubernetes.io/projected/cd554e69-a448-4d73-885a-d1ffcc724c56-kube-api-access-dk997\") pod \"payload-pre-processing-e2e-extmod-b-e3eb98-798ccfff95-kpvst\" (UID: \"cd554e69-a448-4d73-885a-d1ffcc724c56\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e3eb98-798ccfff95-kpvst"
Oct 30 13:37:01.295149 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:01.295100    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd554e69-a448-4d73-885a-d1ffcc724c56-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-e3eb98-798ccfff95-kpvst\" (UID: \"cd554e69-a448-4d73-885a-d1ffcc724c56\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e3eb98-798ccfff95-kpvst"
Oct 30 13:37:01.296541 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:01.296226    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd554e69-a448-4d73-885a-d1ffcc724c56-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-e3eb98-798ccfff95-kpvst\" (UID: \"cd554e69-a448-4d73-885a-d1ffcc724c56\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e3eb98-798ccfff95-kpvst"
Oct 30 13:37:01.330365 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:01.329794    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk997\" (UniqueName: \"kubernetes.io/projected/cd554e69-a448-4d73-885a-d1ffcc724c56-kube-api-access-dk997\") pod \"payload-pre-processing-e2e-extmod-b-e3eb98-798ccfff95-kpvst\" (UID: \"cd554e69-a448-4d73-885a-d1ffcc724c56\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e3eb98-798ccfff95-kpvst"
Oct 30 13:37:01.396940 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:01.396901    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e3eb98-798ccfff95-kpvst"
Oct 30 13:37:01.718970 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:01.718937    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e3eb98-798ccfff95-kpvst"]
Oct 30 13:37:01.720447 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:37:01.720415    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd554e69_a448_4d73_885a_d1ffcc724c56.slice/crio-5e0957dcf07766b9e12fe9a163d82e569639dd84b8e145baa2bb71839b828309 WatchSource:0}: Error finding container 5e0957dcf07766b9e12fe9a163d82e569639dd84b8e145baa2bb71839b828309: Status 404 returned error can't find the container with id 5e0957dcf07766b9e12fe9a163d82e569639dd84b8e145baa2bb71839b828309
Oct 30 13:37:02.734580 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:02.734536    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e3eb98-798ccfff95-kpvst" event={"ID":"cd554e69-a448-4d73-885a-d1ffcc724c56","Type":"ContainerStarted","Data":"e7a4ce73945cd8cc622d77f8046b94243f11a3d48b172fa45be6664297bb2ca6"}
Oct 30 13:37:02.735049 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:02.734594    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e3eb98-798ccfff95-kpvst" event={"ID":"cd554e69-a448-4d73-885a-d1ffcc724c56","Type":"ContainerStarted","Data":"5e0957dcf07766b9e12fe9a163d82e569639dd84b8e145baa2bb71839b828309"}
Oct 30 13:37:02.735049 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:02.734625    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e3eb98-798ccfff95-kpvst"
Oct 30 13:37:02.777605 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:02.777546    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e3eb98-798ccfff95-kpvst" podStartSLOduration=1.7775317510000002 podStartE2EDuration="1.777531751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:37:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:02.775063232 +0000 UTC m=+2041.922330475" watchObservedRunningTime="2026-10-30 13:37:02.777531751 +0000 UTC m=+2041.924798992"
Oct 30 13:37:13.762208 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:13.761866    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e3eb98-798ccfff95-kpvst"
Oct 30 13:37:14.063936 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:14.063831    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-29a3f3/auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc" podUID="a8f4bffb-6519-4c67-bb97-115403fafa6c" containerName="main" containerID="cri-o://7d78fcfb7f42c1f567c34ab6d5c9a07124650d5438379c87d5b3b41983ccf9e7" gracePeriod=45
Oct 30 13:37:14.398551 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:14.398524    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-29a3f3/auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc"
Oct 30 13:37:14.496990 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:14.496950    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8f4bffb-6519-4c67-bb97-115403fafa6c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8f4bffb-6519-4c67-bb97-115403fafa6c-dshm\") pod \"a8f4bffb-6519-4c67-bb97-115403fafa6c\" (UID: \"a8f4bffb-6519-4c67-bb97-115403fafa6c\") "
Oct 30 13:37:14.497280 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:14.497256    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8f4bffb-6519-4c67-bb97-115403fafa6c-tls-certs\" (UniqueName: \"kubernetes.io/secret/a8f4bffb-6519-4c67-bb97-115403fafa6c-tls-certs\") pod \"a8f4bffb-6519-4c67-bb97-115403fafa6c\" (UID: \"a8f4bffb-6519-4c67-bb97-115403fafa6c\") "
Oct 30 13:37:14.497490 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:14.497471    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8f4bffb-6519-4c67-bb97-115403fafa6c-kube-api-access-qlbtw\" (UniqueName: \"kubernetes.io/projected/a8f4bffb-6519-4c67-bb97-115403fafa6c-kube-api-access-qlbtw\") pod \"a8f4bffb-6519-4c67-bb97-115403fafa6c\" (UID: \"a8f4bffb-6519-4c67-bb97-115403fafa6c\") "
Oct 30 13:37:14.497691 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:14.497672    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8f4bffb-6519-4c67-bb97-115403fafa6c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8f4bffb-6519-4c67-bb97-115403fafa6c-tmp-dir\") pod \"a8f4bffb-6519-4c67-bb97-115403fafa6c\" (UID: \"a8f4bffb-6519-4c67-bb97-115403fafa6c\") "
Oct 30 13:37:14.497930 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:14.497913    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8f4bffb-6519-4c67-bb97-115403fafa6c-home\" (UniqueName: \"kubernetes.io/empty-dir/a8f4bffb-6519-4c67-bb97-115403fafa6c-home\") pod \"a8f4bffb-6519-4c67-bb97-115403fafa6c\" (UID: \"a8f4bffb-6519-4c67-bb97-115403fafa6c\") "
Oct 30 13:37:14.498239 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:14.498216    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8f4bffb-6519-4c67-bb97-115403fafa6c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8f4bffb-6519-4c67-bb97-115403fafa6c-model-cache\") pod \"a8f4bffb-6519-4c67-bb97-115403fafa6c\" (UID: \"a8f4bffb-6519-4c67-bb97-115403fafa6c\") "
Oct 30 13:37:14.498387 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:14.498354    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8f4bffb-6519-4c67-bb97-115403fafa6c-tmp-dir" pod "a8f4bffb-6519-4c67-bb97-115403fafa6c" (UID: "a8f4bffb-6519-4c67-bb97-115403fafa6c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:14.498466 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:14.498386    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8f4bffb-6519-4c67-bb97-115403fafa6c-home" pod "a8f4bffb-6519-4c67-bb97-115403fafa6c" (UID: "a8f4bffb-6519-4c67-bb97-115403fafa6c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:14.498758 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:14.498709    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8f4bffb-6519-4c67-bb97-115403fafa6c-model-cache" pod "a8f4bffb-6519-4c67-bb97-115403fafa6c" (UID: "a8f4bffb-6519-4c67-bb97-115403fafa6c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:14.499510 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:14.499489    5093 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8f4bffb-6519-4c67-bb97-115403fafa6c-home\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:37:14.499612 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:14.499599    5093 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8f4bffb-6519-4c67-bb97-115403fafa6c-model-cache\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:37:14.499704 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:14.499693    5093 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8f4bffb-6519-4c67-bb97-115403fafa6c-tmp-dir\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:37:14.500296 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:14.500272    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8f4bffb-6519-4c67-bb97-115403fafa6c-dshm" pod "a8f4bffb-6519-4c67-bb97-115403fafa6c" (UID: "a8f4bffb-6519-4c67-bb97-115403fafa6c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:14.501613 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:14.501588    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8f4bffb-6519-4c67-bb97-115403fafa6c-tls-certs" pod "a8f4bffb-6519-4c67-bb97-115403fafa6c" (UID: "a8f4bffb-6519-4c67-bb97-115403fafa6c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:37:14.501613 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:14.501593    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8f4bffb-6519-4c67-bb97-115403fafa6c-kube-api-access-qlbtw" pod "a8f4bffb-6519-4c67-bb97-115403fafa6c" (UID: "a8f4bffb-6519-4c67-bb97-115403fafa6c"). InnerVolumeSpecName "kube-api-access-qlbtw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:37:14.601479 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:14.601443    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlbtw\" (UniqueName: \"kubernetes.io/projected/a8f4bffb-6519-4c67-bb97-115403fafa6c-kube-api-access-qlbtw\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:37:14.601479 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:14.601474    5093 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8f4bffb-6519-4c67-bb97-115403fafa6c-dshm\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:37:14.601698 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:14.601484    5093 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8f4bffb-6519-4c67-bb97-115403fafa6c-tls-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:37:14.896534 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:14.896496    5093 generic.go:356] "Generic (PLEG): container finished" podID="a8f4bffb-6519-4c67-bb97-115403fafa6c" containerID="7d78fcfb7f42c1f567c34ab6d5c9a07124650d5438379c87d5b3b41983ccf9e7" exitCode=0
Oct 30 13:37:14.897007 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:14.896554    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-29a3f3/auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc"
Oct 30 13:37:14.897007 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:14.896581    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-29a3f3/auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc" event={"ID":"a8f4bffb-6519-4c67-bb97-115403fafa6c","Type":"ContainerDied","Data":"7d78fcfb7f42c1f567c34ab6d5c9a07124650d5438379c87d5b3b41983ccf9e7"}
Oct 30 13:37:14.897007 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:14.896629    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-29a3f3/auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc" event={"ID":"a8f4bffb-6519-4c67-bb97-115403fafa6c","Type":"ContainerDied","Data":"84d1cdaaec8804ded44a8574a56c0d368f001476e08850a89fa7b39040519d1e"}
Oct 30 13:37:14.897007 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:14.896648    5093 scope.go:122] "RemoveContainer" containerID="7d78fcfb7f42c1f567c34ab6d5c9a07124650d5438379c87d5b3b41983ccf9e7"
Oct 30 13:37:14.908228 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:14.908209    5093 scope.go:122] "RemoveContainer" containerID="7d78fcfb7f42c1f567c34ab6d5c9a07124650d5438379c87d5b3b41983ccf9e7"
Oct 30 13:37:14.908527 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:37:14.908506    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d78fcfb7f42c1f567c34ab6d5c9a07124650d5438379c87d5b3b41983ccf9e7\": container with ID starting with 7d78fcfb7f42c1f567c34ab6d5c9a07124650d5438379c87d5b3b41983ccf9e7 not found: ID does not exist" containerID="7d78fcfb7f42c1f567c34ab6d5c9a07124650d5438379c87d5b3b41983ccf9e7"
Oct 30 13:37:14.908595 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:14.908541    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d78fcfb7f42c1f567c34ab6d5c9a07124650d5438379c87d5b3b41983ccf9e7"} err="failed to get container status \"7d78fcfb7f42c1f567c34ab6d5c9a07124650d5438379c87d5b3b41983ccf9e7\": rpc error: code = NotFound desc = could not find container \"7d78fcfb7f42c1f567c34ab6d5c9a07124650d5438379c87d5b3b41983ccf9e7\": container with ID starting with 7d78fcfb7f42c1f567c34ab6d5c9a07124650d5438379c87d5b3b41983ccf9e7 not found: ID does not exist"
Oct 30 13:37:14.962423 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:14.962379    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-29a3f3/auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc"]
Oct 30 13:37:14.974061 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:14.974023    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-29a3f3/auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc"]
Oct 30 13:37:15.003028 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:15.002983    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8f4bffb-6519-4c67-bb97-115403fafa6c" path="/var/lib/kubelet/pods/a8f4bffb-6519-4c67-bb97-115403fafa6c/volumes"
Oct 30 13:37:15.303228 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:15.303190    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-29a3f3/auth-test-model-29a3f3-kserve-648cfd4c7-zhgsc" podUID="a8f4bffb-6519-4c67-bb97-115403fafa6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.142:8000/ready\": context deadline exceeded"
Oct 30 13:37:16.008155 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:15.999641    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm"]
Oct 30 13:37:16.035156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:16.026674    5093 cpu_manager.go:414] "containerMap: removing container" podUID="a8f4bffb-6519-4c67-bb97-115403fafa6c" containerName="main"
Oct 30 13:37:16.035156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:16.026705    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8f4bffb-6519-4c67-bb97-115403fafa6c" containerName="main"
Oct 30 13:37:16.035156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:16.026906    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8f4bffb-6519-4c67-bb97-115403fafa6c" containerName="main"
Oct 30 13:37:16.035156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:16.032898    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm"
Oct 30 13:37:16.045364 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:16.039146    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm"]
Oct 30 13:37:16.053161 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:16.051162    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-74589c-openshift-default-dockercfg-2w2h6\""
Oct 30 13:37:16.159214 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:16.155896    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0d3d9d3-593a-478d-963b-d8b4b395a661-istio-envoy\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm\" (UID: \"b0d3d9d3-593a-478d-963b-d8b4b395a661\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm"
Oct 30 13:37:16.159214 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:16.156035    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkkt7\" (UniqueName: \"kubernetes.io/projected/b0d3d9d3-593a-478d-963b-d8b4b395a661-kube-api-access-nkkt7\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm\" (UID: \"b0d3d9d3-593a-478d-963b-d8b4b395a661\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm"
Oct 30 13:37:16.167154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:16.163296    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0d3d9d3-593a-478d-963b-d8b4b395a661-istio-data\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm\" (UID: \"b0d3d9d3-593a-478d-963b-d8b4b395a661\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm"
Oct 30 13:37:16.167154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:16.163426    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0d3d9d3-593a-478d-963b-d8b4b395a661-istiod-ca-cert\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm\" (UID: \"b0d3d9d3-593a-478d-963b-d8b4b395a661\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm"
Oct 30 13:37:16.167154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:16.163478    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0d3d9d3-593a-478d-963b-d8b4b395a661-workload-socket\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm\" (UID: \"b0d3d9d3-593a-478d-963b-d8b4b395a661\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm"
Oct 30 13:37:16.167154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:16.163565    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0d3d9d3-593a-478d-963b-d8b4b395a661-istio-token\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm\" (UID: \"b0d3d9d3-593a-478d-963b-d8b4b395a661\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm"
Oct 30 13:37:16.167154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:16.163618    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0d3d9d3-593a-478d-963b-d8b4b395a661-istio-podinfo\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm\" (UID: \"b0d3d9d3-593a-478d-963b-d8b4b395a661\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm"
Oct 30 13:37:16.167154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:16.163654    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0d3d9d3-593a-478d-963b-d8b4b395a661-workload-certs\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm\" (UID: \"b0d3d9d3-593a-478d-963b-d8b4b395a661\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm"
Oct 30 13:37:16.167154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:16.163725    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0d3d9d3-593a-478d-963b-d8b4b395a661-credential-socket\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm\" (UID: \"b0d3d9d3-593a-478d-963b-d8b4b395a661\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm"
Oct 30 13:37:16.270533 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:16.270429    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0d3d9d3-593a-478d-963b-d8b4b395a661-istio-envoy\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm\" (UID: \"b0d3d9d3-593a-478d-963b-d8b4b395a661\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm"
Oct 30 13:37:16.286155 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:16.271269    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkkt7\" (UniqueName: \"kubernetes.io/projected/b0d3d9d3-593a-478d-963b-d8b4b395a661-kube-api-access-nkkt7\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm\" (UID: \"b0d3d9d3-593a-478d-963b-d8b4b395a661\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm"
Oct 30 13:37:16.286346 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:16.286170    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0d3d9d3-593a-478d-963b-d8b4b395a661-istio-data\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm\" (UID: \"b0d3d9d3-593a-478d-963b-d8b4b395a661\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm"
Oct 30 13:37:16.286346 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:16.286239    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0d3d9d3-593a-478d-963b-d8b4b395a661-istiod-ca-cert\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm\" (UID: \"b0d3d9d3-593a-478d-963b-d8b4b395a661\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm"
Oct 30 13:37:16.286346 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:16.286286    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0d3d9d3-593a-478d-963b-d8b4b395a661-workload-socket\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm\" (UID: \"b0d3d9d3-593a-478d-963b-d8b4b395a661\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm"
Oct 30 13:37:16.286516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:16.286380    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0d3d9d3-593a-478d-963b-d8b4b395a661-istio-token\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm\" (UID: \"b0d3d9d3-593a-478d-963b-d8b4b395a661\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm"
Oct 30 13:37:16.286516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:16.286442    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0d3d9d3-593a-478d-963b-d8b4b395a661-istio-podinfo\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm\" (UID: \"b0d3d9d3-593a-478d-963b-d8b4b395a661\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm"
Oct 30 13:37:16.286516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:16.286483    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0d3d9d3-593a-478d-963b-d8b4b395a661-workload-certs\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm\" (UID: \"b0d3d9d3-593a-478d-963b-d8b4b395a661\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm"
Oct 30 13:37:16.286650 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:16.286558    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0d3d9d3-593a-478d-963b-d8b4b395a661-credential-socket\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm\" (UID: \"b0d3d9d3-593a-478d-963b-d8b4b395a661\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm"
Oct 30 13:37:16.288388 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:16.286998    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0d3d9d3-593a-478d-963b-d8b4b395a661-credential-socket\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm\" (UID: \"b0d3d9d3-593a-478d-963b-d8b4b395a661\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm"
Oct 30 13:37:16.288388 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:16.285543    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0d3d9d3-593a-478d-963b-d8b4b395a661-istio-envoy\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm\" (UID: \"b0d3d9d3-593a-478d-963b-d8b4b395a661\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm"
Oct 30 13:37:16.288388 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:16.287362    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0d3d9d3-593a-478d-963b-d8b4b395a661-istio-data\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm\" (UID: \"b0d3d9d3-593a-478d-963b-d8b4b395a661\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm"
Oct 30 13:37:16.288388 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:16.287954    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0d3d9d3-593a-478d-963b-d8b4b395a661-istiod-ca-cert\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm\" (UID: \"b0d3d9d3-593a-478d-963b-d8b4b395a661\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm"
Oct 30 13:37:16.288388 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:16.288235    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0d3d9d3-593a-478d-963b-d8b4b395a661-workload-socket\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm\" (UID: \"b0d3d9d3-593a-478d-963b-d8b4b395a661\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm"
Oct 30 13:37:16.288759 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:16.288485    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0d3d9d3-593a-478d-963b-d8b4b395a661-workload-certs\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm\" (UID: \"b0d3d9d3-593a-478d-963b-d8b4b395a661\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm"
Oct 30 13:37:16.291871 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:16.291822    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0d3d9d3-593a-478d-963b-d8b4b395a661-istio-podinfo\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm\" (UID: \"b0d3d9d3-593a-478d-963b-d8b4b395a661\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm"
Oct 30 13:37:16.306701 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:16.306674    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkkt7\" (UniqueName: \"kubernetes.io/projected/b0d3d9d3-593a-478d-963b-d8b4b395a661-kube-api-access-nkkt7\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm\" (UID: \"b0d3d9d3-593a-478d-963b-d8b4b395a661\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm"
Oct 30 13:37:16.310267 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:16.310222    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0d3d9d3-593a-478d-963b-d8b4b395a661-istio-token\") pod \"e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm\" (UID: \"b0d3d9d3-593a-478d-963b-d8b4b395a661\") " pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm"
Oct 30 13:37:16.415273 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:16.415237    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm"
Oct 30 13:37:16.776995 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:37:16.776965    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0d3d9d3_593a_478d_963b_d8b4b395a661.slice/crio-e488a8f5502f02bab02f075053473ed243c5c8ecf184cfdaa4ff0ec7a3e85e64 WatchSource:0}: Error finding container e488a8f5502f02bab02f075053473ed243c5c8ecf184cfdaa4ff0ec7a3e85e64: Status 404 returned error can't find the container with id e488a8f5502f02bab02f075053473ed243c5c8ecf184cfdaa4ff0ec7a3e85e64
Oct 30 13:37:16.778207 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:16.778178    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm"]
Oct 30 13:37:16.779953 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:16.779919    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:37:16.780025 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:16.779985    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:37:16.780025 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:16.780014    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:37:16.917514 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:16.917481    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm" event={"ID":"b0d3d9d3-593a-478d-963b-d8b4b395a661","Type":"ContainerStarted","Data":"b3b733ee51a45db343a28e5099a1558c7507df79b6084d95825f8c769dc461d3"}
Oct 30 13:37:16.917678 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:16.917536    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm" event={"ID":"b0d3d9d3-593a-478d-963b-d8b4b395a661","Type":"ContainerStarted","Data":"e488a8f5502f02bab02f075053473ed243c5c8ecf184cfdaa4ff0ec7a3e85e64"}
Oct 30 13:37:16.973772 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:16.973713    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm" podStartSLOduration=1.9736924249999999 podStartE2EDuration="1.973692425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:37:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:16.971612393 +0000 UTC m=+2056.118879638" watchObservedRunningTime="2026-10-30 13:37:16.973692425 +0000 UTC m=+2056.120959672"
Oct 30 13:37:17.415823 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:17.415787    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm"
Oct 30 13:37:17.417927 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:17.417895    5093 patch_prober.go:29] interesting pod/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 30 13:37:17.418072 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:17.417942    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm" podUID="b0d3d9d3-593a-478d-963b-d8b4b395a661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 30 13:37:17.690372 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:17.690335    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-671403/noten-model-671403-kserve-797fc96b45-sslm8"]
Oct 30 13:37:17.695235 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:17.695204    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-671403/noten-model-671403-kserve-797fc96b45-sslm8"
Oct 30 13:37:17.711048 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:17.710485    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d889b21-5eaf-47f1-b298-850ad1306936-model-cache\") pod \"noten-model-671403-kserve-797fc96b45-sslm8\" (UID: \"3d889b21-5eaf-47f1-b298-850ad1306936\") " pod="ai-tenant-e2e-noten-671403/noten-model-671403-kserve-797fc96b45-sslm8"
Oct 30 13:37:17.711048 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:17.710542    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d889b21-5eaf-47f1-b298-850ad1306936-home\") pod \"noten-model-671403-kserve-797fc96b45-sslm8\" (UID: \"3d889b21-5eaf-47f1-b298-850ad1306936\") " pod="ai-tenant-e2e-noten-671403/noten-model-671403-kserve-797fc96b45-sslm8"
Oct 30 13:37:17.711048 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:17.710574    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d889b21-5eaf-47f1-b298-850ad1306936-dshm\") pod \"noten-model-671403-kserve-797fc96b45-sslm8\" (UID: \"3d889b21-5eaf-47f1-b298-850ad1306936\") " pod="ai-tenant-e2e-noten-671403/noten-model-671403-kserve-797fc96b45-sslm8"
Oct 30 13:37:17.711048 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:17.710870    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sddqw\" (UniqueName: \"kubernetes.io/projected/3d889b21-5eaf-47f1-b298-850ad1306936-kube-api-access-sddqw\") pod \"noten-model-671403-kserve-797fc96b45-sslm8\" (UID: \"3d889b21-5eaf-47f1-b298-850ad1306936\") " pod="ai-tenant-e2e-noten-671403/noten-model-671403-kserve-797fc96b45-sslm8"
Oct 30 13:37:17.711048 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:17.710980    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d889b21-5eaf-47f1-b298-850ad1306936-tls-certs\") pod \"noten-model-671403-kserve-797fc96b45-sslm8\" (UID: \"3d889b21-5eaf-47f1-b298-850ad1306936\") " pod="ai-tenant-e2e-noten-671403/noten-model-671403-kserve-797fc96b45-sslm8"
Oct 30 13:37:17.729271 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:17.720148    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-671403/noten-model-671403-kserve-797fc96b45-sslm8"]
Oct 30 13:37:17.729271 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:17.711031    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d889b21-5eaf-47f1-b298-850ad1306936-tmp-dir\") pod \"noten-model-671403-kserve-797fc96b45-sslm8\" (UID: \"3d889b21-5eaf-47f1-b298-850ad1306936\") " pod="ai-tenant-e2e-noten-671403/noten-model-671403-kserve-797fc96b45-sslm8"
Oct 30 13:37:17.729271 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:17.721256    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-671403\"/\"kube-root-ca.crt\""
Oct 30 13:37:17.729271 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:17.721422    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-671403\"/\"openshift-service-ca.crt\""
Oct 30 13:37:17.729271 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:17.724757    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-671403\"/\"default-dockercfg-m9mwf\""
Oct 30 13:37:17.769318 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:17.769279    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-671403\"/\"noten-model-671403-kserve-self-signed-certs\""
Oct 30 13:37:17.822950 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:17.822913    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sddqw\" (UniqueName: \"kubernetes.io/projected/3d889b21-5eaf-47f1-b298-850ad1306936-kube-api-access-sddqw\") pod \"noten-model-671403-kserve-797fc96b45-sslm8\" (UID: \"3d889b21-5eaf-47f1-b298-850ad1306936\") " pod="ai-tenant-e2e-noten-671403/noten-model-671403-kserve-797fc96b45-sslm8"
Oct 30 13:37:17.823161 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:17.823009    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d889b21-5eaf-47f1-b298-850ad1306936-tls-certs\") pod \"noten-model-671403-kserve-797fc96b45-sslm8\" (UID: \"3d889b21-5eaf-47f1-b298-850ad1306936\") " pod="ai-tenant-e2e-noten-671403/noten-model-671403-kserve-797fc96b45-sslm8"
Oct 30 13:37:17.823161 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:17.823050    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d889b21-5eaf-47f1-b298-850ad1306936-tmp-dir\") pod \"noten-model-671403-kserve-797fc96b45-sslm8\" (UID: \"3d889b21-5eaf-47f1-b298-850ad1306936\") " pod="ai-tenant-e2e-noten-671403/noten-model-671403-kserve-797fc96b45-sslm8"
Oct 30 13:37:17.823161 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:17.823156    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d889b21-5eaf-47f1-b298-850ad1306936-model-cache\") pod \"noten-model-671403-kserve-797fc96b45-sslm8\" (UID: \"3d889b21-5eaf-47f1-b298-850ad1306936\") " pod="ai-tenant-e2e-noten-671403/noten-model-671403-kserve-797fc96b45-sslm8"
Oct 30 13:37:17.823350 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:17.823182    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d889b21-5eaf-47f1-b298-850ad1306936-home\") pod \"noten-model-671403-kserve-797fc96b45-sslm8\" (UID: \"3d889b21-5eaf-47f1-b298-850ad1306936\") " pod="ai-tenant-e2e-noten-671403/noten-model-671403-kserve-797fc96b45-sslm8"
Oct 30 13:37:17.823350 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:17.823196    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d889b21-5eaf-47f1-b298-850ad1306936-dshm\") pod \"noten-model-671403-kserve-797fc96b45-sslm8\" (UID: \"3d889b21-5eaf-47f1-b298-850ad1306936\") " pod="ai-tenant-e2e-noten-671403/noten-model-671403-kserve-797fc96b45-sslm8"
Oct 30 13:37:17.824587 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:17.824486    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d889b21-5eaf-47f1-b298-850ad1306936-tmp-dir\") pod \"noten-model-671403-kserve-797fc96b45-sslm8\" (UID: \"3d889b21-5eaf-47f1-b298-850ad1306936\") " pod="ai-tenant-e2e-noten-671403/noten-model-671403-kserve-797fc96b45-sslm8"
Oct 30 13:37:17.824744 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:17.824646    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d889b21-5eaf-47f1-b298-850ad1306936-model-cache\") pod \"noten-model-671403-kserve-797fc96b45-sslm8\" (UID: \"3d889b21-5eaf-47f1-b298-850ad1306936\") " pod="ai-tenant-e2e-noten-671403/noten-model-671403-kserve-797fc96b45-sslm8"
Oct 30 13:37:17.824805 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:17.824735    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d889b21-5eaf-47f1-b298-850ad1306936-home\") pod \"noten-model-671403-kserve-797fc96b45-sslm8\" (UID: \"3d889b21-5eaf-47f1-b298-850ad1306936\") " pod="ai-tenant-e2e-noten-671403/noten-model-671403-kserve-797fc96b45-sslm8"
Oct 30 13:37:17.869529 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:17.869431    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d889b21-5eaf-47f1-b298-850ad1306936-tls-certs\") pod \"noten-model-671403-kserve-797fc96b45-sslm8\" (UID: \"3d889b21-5eaf-47f1-b298-850ad1306936\") " pod="ai-tenant-e2e-noten-671403/noten-model-671403-kserve-797fc96b45-sslm8"
Oct 30 13:37:17.870588 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:17.870560    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d889b21-5eaf-47f1-b298-850ad1306936-dshm\") pod \"noten-model-671403-kserve-797fc96b45-sslm8\" (UID: \"3d889b21-5eaf-47f1-b298-850ad1306936\") " pod="ai-tenant-e2e-noten-671403/noten-model-671403-kserve-797fc96b45-sslm8"
Oct 30 13:37:17.871978 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:17.871955    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sddqw\" (UniqueName: \"kubernetes.io/projected/3d889b21-5eaf-47f1-b298-850ad1306936-kube-api-access-sddqw\") pod \"noten-model-671403-kserve-797fc96b45-sslm8\" (UID: \"3d889b21-5eaf-47f1-b298-850ad1306936\") " pod="ai-tenant-e2e-noten-671403/noten-model-671403-kserve-797fc96b45-sslm8"
Oct 30 13:37:18.082641 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:18.082532    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-671403/noten-model-671403-kserve-797fc96b45-sslm8"
Oct 30 13:37:18.418637 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:18.418598    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-671403/noten-model-671403-kserve-797fc96b45-sslm8"]
Oct 30 13:37:18.418979 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:37:18.418643    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d889b21_5eaf_47f1_b298_850ad1306936.slice/crio-5c97b9f89e304efe60b2e70cdf1302720c685bdf69550abd621fd4cce752bbe5 WatchSource:0}: Error finding container 5c97b9f89e304efe60b2e70cdf1302720c685bdf69550abd621fd4cce752bbe5: Status 404 returned error can't find the container with id 5c97b9f89e304efe60b2e70cdf1302720c685bdf69550abd621fd4cce752bbe5
Oct 30 13:37:18.422332 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:18.422305    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm"
Oct 30 13:37:19.021111 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:19.021068    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-671403/noten-model-671403-kserve-797fc96b45-sslm8" event={"ID":"3d889b21-5eaf-47f1-b298-850ad1306936","Type":"ContainerStarted","Data":"1101f0c8d481985088ba2a2bb9ab794b08a5f2af3e44581dbe0c2bbf956051b6"}
Oct 30 13:37:19.021298 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:19.021123    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-671403/noten-model-671403-kserve-797fc96b45-sslm8" event={"ID":"3d889b21-5eaf-47f1-b298-850ad1306936","Type":"ContainerStarted","Data":"5c97b9f89e304efe60b2e70cdf1302720c685bdf69550abd621fd4cce752bbe5"}
Oct 30 13:37:19.021625 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:19.021607    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm"
Oct 30 13:37:19.022693 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:19.022669    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm"
Oct 30 13:37:19.064956 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:19.064887    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-671403/noten-model-671403-kserve-797fc96b45-sslm8" podStartSLOduration=2.064868476 podStartE2EDuration="2.064868476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:37:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:19.062384327 +0000 UTC m=+2058.209651569" watchObservedRunningTime="2026-10-30 13:37:19.064868476 +0000 UTC m=+2058.212135718"
Oct 30 13:37:19.082929 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:19.082895    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-671403/noten-model-671403-kserve-797fc96b45-sslm8"
Oct 30 13:37:21.181792 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:37:21.181755    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:37:21.671763 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:21.671708    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-671403/noten-model-671403-kserve-797fc96b45-sslm8"]
Oct 30 13:37:24.115482 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:24.115450    5093 generic.go:356] "Generic (PLEG): container finished" podID="8285d6ec-8ad7-4a33-a696-d32242da2f80" containerID="58cb4ccd0578760096ff3f24e0e118b4a3cbe5cdc22db71b7f3feb99caa1207b" exitCode=0
Oct 30 13:37:24.116053 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:24.115527    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp" event={"ID":"8285d6ec-8ad7-4a33-a696-d32242da2f80","Type":"ContainerDied","Data":"58cb4ccd0578760096ff3f24e0e118b4a3cbe5cdc22db71b7f3feb99caa1207b"}
Oct 30 13:37:24.116151 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:24.116074    5093 scope.go:122] "RemoveContainer" containerID="58cb4ccd0578760096ff3f24e0e118b4a3cbe5cdc22db71b7f3feb99caa1207b"
Oct 30 13:37:24.117959 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:24.117928    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:37:24.118060 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:24.118016    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:37:24.118060 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:24.118051    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:37:25.144155 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:25.138627    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp" event={"ID":"8285d6ec-8ad7-4a33-a696-d32242da2f80","Type":"ContainerStarted","Data":"cdce5bd18be49daaa841dafa956db4563e4eecd8bdb0332ad23cf63621d91484"}
Oct 30 13:37:25.781645 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:25.781607    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp"
Oct 30 13:37:25.782883 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:25.782853    5093 patch_prober.go:29] interesting pod/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Oct 30 13:37:25.783010 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:25.782908    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp" podUID="8285d6ec-8ad7-4a33-a696-d32242da2f80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Oct 30 13:37:26.782147 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:26.782097    5093 patch_prober.go:29] interesting pod/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Oct 30 13:37:26.782667 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:26.782175    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp" podUID="8285d6ec-8ad7-4a33-a696-d32242da2f80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Oct 30 13:37:27.781858 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:27.781815    5093 patch_prober.go:29] interesting pod/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Oct 30 13:37:27.782079 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:27.781900    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp" podUID="8285d6ec-8ad7-4a33-a696-d32242da2f80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Oct 30 13:37:28.782164 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:28.782082    5093 patch_prober.go:29] interesting pod/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Oct 30 13:37:28.782708 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:28.782216    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp" podUID="8285d6ec-8ad7-4a33-a696-d32242da2f80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Oct 30 13:37:29.783298 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:29.782858    5093 patch_prober.go:29] interesting pod/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Oct 30 13:37:29.783298 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:29.782925    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp" podUID="8285d6ec-8ad7-4a33-a696-d32242da2f80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Oct 30 13:37:30.046362 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:30.046252    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp"
Oct 30 13:37:30.788300 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:30.788266    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp"
Oct 30 13:37:30.789520 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:30.789493    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp"
Oct 30 13:37:36.501560 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:36.501524    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3eb98-7789bfbbbc-c4n45"]
Oct 30 13:37:36.506565 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:36.506541    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3eb98-7789bfbbbc-c4n45"
Oct 30 13:37:36.525877 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:36.525753    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3eb98-7789bfbbbc-c4n45"]
Oct 30 13:37:36.533583 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:36.533553    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-e3eb98-serving-cert\""
Oct 30 13:37:36.580962 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:36.580922    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-e3eb98-7789bfbbbc-c4n45\" (UID: \"c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3eb98-7789bfbbbc-c4n45"
Oct 30 13:37:36.581409 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:36.581368    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd-maas-api-tls\") pod \"maas-api-e2e-extmod-b-e3eb98-7789bfbbbc-c4n45\" (UID: \"c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3eb98-7789bfbbbc-c4n45"
Oct 30 13:37:36.581520 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:36.581477    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmghd\" (UniqueName: \"kubernetes.io/projected/c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd-kube-api-access-nmghd\") pod \"maas-api-e2e-extmod-b-e3eb98-7789bfbbbc-c4n45\" (UID: \"c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3eb98-7789bfbbbc-c4n45"
Oct 30 13:37:36.581691 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:36.581667    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd-metrics-certs\") pod \"maas-api-e2e-extmod-b-e3eb98-7789bfbbbc-c4n45\" (UID: \"c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3eb98-7789bfbbbc-c4n45"
Oct 30 13:37:36.682674 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:36.682633    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmghd\" (UniqueName: \"kubernetes.io/projected/c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd-kube-api-access-nmghd\") pod \"maas-api-e2e-extmod-b-e3eb98-7789bfbbbc-c4n45\" (UID: \"c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3eb98-7789bfbbbc-c4n45"
Oct 30 13:37:36.682876 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:36.682718    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd-metrics-certs\") pod \"maas-api-e2e-extmod-b-e3eb98-7789bfbbbc-c4n45\" (UID: \"c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3eb98-7789bfbbbc-c4n45"
Oct 30 13:37:36.682876 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:36.682820    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-e3eb98-7789bfbbbc-c4n45\" (UID: \"c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3eb98-7789bfbbbc-c4n45"
Oct 30 13:37:36.682991 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:36.682966    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd-maas-api-tls\") pod \"maas-api-e2e-extmod-b-e3eb98-7789bfbbbc-c4n45\" (UID: \"c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3eb98-7789bfbbbc-c4n45"
Oct 30 13:37:36.683578 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:36.683543    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-e3eb98-7789bfbbbc-c4n45\" (UID: \"c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3eb98-7789bfbbbc-c4n45"
Oct 30 13:37:36.685310 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:36.685287    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd-metrics-certs\") pod \"maas-api-e2e-extmod-b-e3eb98-7789bfbbbc-c4n45\" (UID: \"c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3eb98-7789bfbbbc-c4n45"
Oct 30 13:37:36.685432 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:36.685413    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd-maas-api-tls\") pod \"maas-api-e2e-extmod-b-e3eb98-7789bfbbbc-c4n45\" (UID: \"c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3eb98-7789bfbbbc-c4n45"
Oct 30 13:37:36.702689 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:36.702652    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmghd\" (UniqueName: \"kubernetes.io/projected/c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd-kube-api-access-nmghd\") pod \"maas-api-e2e-extmod-b-e3eb98-7789bfbbbc-c4n45\" (UID: \"c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3eb98-7789bfbbbc-c4n45"
Oct 30 13:37:36.823780 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:36.823681    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3eb98-7789bfbbbc-c4n45"
Oct 30 13:37:37.119889 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:37.119789    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-671403/noten-model-671403-kserve-797fc96b45-sslm8" podUID="3d889b21-5eaf-47f1-b298-850ad1306936" containerName="main" containerID="cri-o://1101f0c8d481985088ba2a2bb9ab794b08a5f2af3e44581dbe0c2bbf956051b6" gracePeriod=45
Oct 30 13:37:37.140629 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:37.140583    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3eb98-7789bfbbbc-c4n45"]
Oct 30 13:37:37.169766 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:37:37.169723    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc04d4128_e9ec_4eeb_8eba_5d6d6c760ccd.slice/crio-d40fd3e2fbdd55d99981af0b0c6ceee53b7ebb9fdb8125997088f85ae38ff4d7 WatchSource:0}: Error finding container d40fd3e2fbdd55d99981af0b0c6ceee53b7ebb9fdb8125997088f85ae38ff4d7: Status 404 returned error can't find the container with id d40fd3e2fbdd55d99981af0b0c6ceee53b7ebb9fdb8125997088f85ae38ff4d7
Oct 30 13:37:37.279852 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:37.279817    5093 generic.go:356] "Generic (PLEG): container finished" podID="3d889b21-5eaf-47f1-b298-850ad1306936" containerID="1101f0c8d481985088ba2a2bb9ab794b08a5f2af3e44581dbe0c2bbf956051b6" exitCode=0
Oct 30 13:37:37.280026 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:37.279887    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-671403/noten-model-671403-kserve-797fc96b45-sslm8" event={"ID":"3d889b21-5eaf-47f1-b298-850ad1306936","Type":"ContainerDied","Data":"1101f0c8d481985088ba2a2bb9ab794b08a5f2af3e44581dbe0c2bbf956051b6"}
Oct 30 13:37:37.281239 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:37.281206    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3eb98-7789bfbbbc-c4n45" event={"ID":"c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd","Type":"ContainerStarted","Data":"d40fd3e2fbdd55d99981af0b0c6ceee53b7ebb9fdb8125997088f85ae38ff4d7"}
Oct 30 13:37:37.395553 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:37.395525    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-671403/noten-model-671403-kserve-797fc96b45-sslm8"
Oct 30 13:37:37.514916 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:37.506597    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d889b21-5eaf-47f1-b298-850ad1306936-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d889b21-5eaf-47f1-b298-850ad1306936-dshm\") pod \"3d889b21-5eaf-47f1-b298-850ad1306936\" (UID: \"3d889b21-5eaf-47f1-b298-850ad1306936\") "
Oct 30 13:37:37.514916 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:37.506801    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d889b21-5eaf-47f1-b298-850ad1306936-home\" (UniqueName: \"kubernetes.io/empty-dir/3d889b21-5eaf-47f1-b298-850ad1306936-home\") pod \"3d889b21-5eaf-47f1-b298-850ad1306936\" (UID: \"3d889b21-5eaf-47f1-b298-850ad1306936\") "
Oct 30 13:37:37.514916 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:37.506874    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d889b21-5eaf-47f1-b298-850ad1306936-kube-api-access-sddqw\" (UniqueName: \"kubernetes.io/projected/3d889b21-5eaf-47f1-b298-850ad1306936-kube-api-access-sddqw\") pod \"3d889b21-5eaf-47f1-b298-850ad1306936\" (UID: \"3d889b21-5eaf-47f1-b298-850ad1306936\") "
Oct 30 13:37:37.514916 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:37.506941    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d889b21-5eaf-47f1-b298-850ad1306936-tls-certs\" (UniqueName: \"kubernetes.io/secret/3d889b21-5eaf-47f1-b298-850ad1306936-tls-certs\") pod \"3d889b21-5eaf-47f1-b298-850ad1306936\" (UID: \"3d889b21-5eaf-47f1-b298-850ad1306936\") "
Oct 30 13:37:37.514916 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:37.507081    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d889b21-5eaf-47f1-b298-850ad1306936-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d889b21-5eaf-47f1-b298-850ad1306936-tmp-dir\") pod \"3d889b21-5eaf-47f1-b298-850ad1306936\" (UID: \"3d889b21-5eaf-47f1-b298-850ad1306936\") "
Oct 30 13:37:37.514916 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:37.507124    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d889b21-5eaf-47f1-b298-850ad1306936-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d889b21-5eaf-47f1-b298-850ad1306936-model-cache\") pod \"3d889b21-5eaf-47f1-b298-850ad1306936\" (UID: \"3d889b21-5eaf-47f1-b298-850ad1306936\") "
Oct 30 13:37:37.518227 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:37.515892    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d889b21-5eaf-47f1-b298-850ad1306936-model-cache" pod "3d889b21-5eaf-47f1-b298-850ad1306936" (UID: "3d889b21-5eaf-47f1-b298-850ad1306936"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:37.578380 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:37.551243    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d889b21-5eaf-47f1-b298-850ad1306936-tmp-dir" pod "3d889b21-5eaf-47f1-b298-850ad1306936" (UID: "3d889b21-5eaf-47f1-b298-850ad1306936"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:37.578380 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:37.551476    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d889b21-5eaf-47f1-b298-850ad1306936-home" pod "3d889b21-5eaf-47f1-b298-850ad1306936" (UID: "3d889b21-5eaf-47f1-b298-850ad1306936"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:37.609374 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:37.609290    5093 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d889b21-5eaf-47f1-b298-850ad1306936-home\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:37:37.609374 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:37.609325    5093 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d889b21-5eaf-47f1-b298-850ad1306936-tmp-dir\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:37:37.609374 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:37.609342    5093 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d889b21-5eaf-47f1-b298-850ad1306936-model-cache\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:37:37.621646 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:37.621469    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d889b21-5eaf-47f1-b298-850ad1306936-tls-certs" pod "3d889b21-5eaf-47f1-b298-850ad1306936" (UID: "3d889b21-5eaf-47f1-b298-850ad1306936"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:37:37.623533 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:37.623479    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d889b21-5eaf-47f1-b298-850ad1306936-kube-api-access-sddqw" pod "3d889b21-5eaf-47f1-b298-850ad1306936" (UID: "3d889b21-5eaf-47f1-b298-850ad1306936"). InnerVolumeSpecName "kube-api-access-sddqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:37:37.638679 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:37.638588    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d889b21-5eaf-47f1-b298-850ad1306936-dshm" pod "3d889b21-5eaf-47f1-b298-850ad1306936" (UID: "3d889b21-5eaf-47f1-b298-850ad1306936"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:37.711000 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:37.710973    5093 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d889b21-5eaf-47f1-b298-850ad1306936-dshm\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:37:37.711000 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:37.711002    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sddqw\" (UniqueName: \"kubernetes.io/projected/3d889b21-5eaf-47f1-b298-850ad1306936-kube-api-access-sddqw\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:37:37.711168 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:37.711012    5093 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d889b21-5eaf-47f1-b298-850ad1306936-tls-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:37:38.298797 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:38.298762    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-671403/noten-model-671403-kserve-797fc96b45-sslm8"
Oct 30 13:37:38.298797 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:38.298784    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-671403/noten-model-671403-kserve-797fc96b45-sslm8" event={"ID":"3d889b21-5eaf-47f1-b298-850ad1306936","Type":"ContainerDied","Data":"5c97b9f89e304efe60b2e70cdf1302720c685bdf69550abd621fd4cce752bbe5"}
Oct 30 13:37:38.299090 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:38.298840    5093 scope.go:122] "RemoveContainer" containerID="1101f0c8d481985088ba2a2bb9ab794b08a5f2af3e44581dbe0c2bbf956051b6"
Oct 30 13:37:38.309335 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:38.309294    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3eb98-7789bfbbbc-c4n45" event={"ID":"c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd","Type":"ContainerStarted","Data":"d9553bd1508bc27465bdbf17d7c047329e29dd0f1bcc9cd832de0c8cbc1a5cf3"}
Oct 30 13:37:38.309599 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:38.309577    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3eb98-7789bfbbbc-c4n45"
Oct 30 13:37:38.388174 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:38.383416    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-671403/noten-model-671403-kserve-797fc96b45-sslm8"]
Oct 30 13:37:38.394384 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:38.394347    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-671403/noten-model-671403-kserve-797fc96b45-sslm8"]
Oct 30 13:37:38.420071 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:38.420004    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3eb98-7789bfbbbc-c4n45" podStartSLOduration=1.9428072859999999 podStartE2EDuration="2.419972458s" totalImagesPullingTime="477.165172ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:37:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:38.418344636 +0000 UTC m=+2077.565611908" watchObservedRunningTime="2026-10-30 13:37:38.419972458 +0000 UTC m=+2077.567239700"
Oct 30 13:37:39.046156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:39.042684    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d889b21-5eaf-47f1-b298-850ad1306936" path="/var/lib/kubelet/pods/3d889b21-5eaf-47f1-b298-850ad1306936/volumes"
Oct 30 13:37:44.324741 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:37:44.324704    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3eb98-7789bfbbbc-c4n45"
Oct 30 13:37:51.169572 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:37:51.169517    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:38:01.457207 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:01.457173    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jdj4_f93ee017-ab79-447c-90e0-5b85b71dba24/console-operator/1.log"
Oct 30 13:38:01.460771 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:01.460745    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jdj4_f93ee017-ab79-447c-90e0-5b85b71dba24/console-operator/1.log"
Oct 30 13:38:01.463759 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:01.463736    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tnm_3a675ac7-fbe1-449b-accf-bb0456ed0bd3/ovn-acl-logging/0.log"
Oct 30 13:38:01.467326 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:01.467300    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tnm_3a675ac7-fbe1-449b-accf-bb0456ed0bd3/ovn-acl-logging/0.log"
Oct 30 13:38:13.755741 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:13.755647    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c3efb2-5f44b7cbcf-4zkz8"]
Oct 30 13:38:13.756976 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:13.756945    5093 cpu_manager.go:414] "containerMap: removing container" podUID="3d889b21-5eaf-47f1-b298-850ad1306936" containerName="main"
Oct 30 13:38:13.756976 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:13.756976    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d889b21-5eaf-47f1-b298-850ad1306936" containerName="main"
Oct 30 13:38:13.757215 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:13.757200    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d889b21-5eaf-47f1-b298-850ad1306936" containerName="main"
Oct 30 13:38:13.763444 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:13.763412    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c3efb2-5f44b7cbcf-4zkz8"
Oct 30 13:38:13.782574 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:13.782540    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c3efb2-5f44b7cbcf-4zkz8"]
Oct 30 13:38:13.791019 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:13.790983    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-c3efb2\""
Oct 30 13:38:13.974152 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:13.950973    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwxzm\" (UniqueName: \"kubernetes.io/projected/21032810-cf90-4fef-816b-d66bf485aefb-kube-api-access-bwxzm\") pod \"payload-pre-processing-e2e-extmod-a-c3efb2-5f44b7cbcf-4zkz8\" (UID: \"21032810-cf90-4fef-816b-d66bf485aefb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c3efb2-5f44b7cbcf-4zkz8"
Oct 30 13:38:14.031280 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:14.025953    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21032810-cf90-4fef-816b-d66bf485aefb-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c3efb2-5f44b7cbcf-4zkz8\" (UID: \"21032810-cf90-4fef-816b-d66bf485aefb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c3efb2-5f44b7cbcf-4zkz8"
Oct 30 13:38:14.127516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:14.127426    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwxzm\" (UniqueName: \"kubernetes.io/projected/21032810-cf90-4fef-816b-d66bf485aefb-kube-api-access-bwxzm\") pod \"payload-pre-processing-e2e-extmod-a-c3efb2-5f44b7cbcf-4zkz8\" (UID: \"21032810-cf90-4fef-816b-d66bf485aefb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c3efb2-5f44b7cbcf-4zkz8"
Oct 30 13:38:14.127738 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:14.127660    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21032810-cf90-4fef-816b-d66bf485aefb-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c3efb2-5f44b7cbcf-4zkz8\" (UID: \"21032810-cf90-4fef-816b-d66bf485aefb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c3efb2-5f44b7cbcf-4zkz8"
Oct 30 13:38:14.128624 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:14.128593    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21032810-cf90-4fef-816b-d66bf485aefb-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c3efb2-5f44b7cbcf-4zkz8\" (UID: \"21032810-cf90-4fef-816b-d66bf485aefb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c3efb2-5f44b7cbcf-4zkz8"
Oct 30 13:38:14.148498 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:14.148463    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwxzm\" (UniqueName: \"kubernetes.io/projected/21032810-cf90-4fef-816b-d66bf485aefb-kube-api-access-bwxzm\") pod \"payload-pre-processing-e2e-extmod-a-c3efb2-5f44b7cbcf-4zkz8\" (UID: \"21032810-cf90-4fef-816b-d66bf485aefb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c3efb2-5f44b7cbcf-4zkz8"
Oct 30 13:38:14.379732 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:14.379624    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c3efb2-5f44b7cbcf-4zkz8"
Oct 30 13:38:14.690185 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:14.687660    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c3efb2-5f44b7cbcf-4zkz8"]
Oct 30 13:38:15.629166 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:15.626678    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c3efb2-5f44b7cbcf-4zkz8" event={"ID":"21032810-cf90-4fef-816b-d66bf485aefb","Type":"ContainerStarted","Data":"02dc8a6107e3d6853c3f2eb19fc4a0b69977f60ff3ae46bed49e604c9df1174c"}
Oct 30 13:38:15.629166 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:15.626765    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c3efb2-5f44b7cbcf-4zkz8" event={"ID":"21032810-cf90-4fef-816b-d66bf485aefb","Type":"ContainerStarted","Data":"1a4735a52158310688eaa398d761978527ba068c28f05ba8297c41572c8e9c6c"}
Oct 30 13:38:15.629166 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:15.628151    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c3efb2-5f44b7cbcf-4zkz8"
Oct 30 13:38:15.683034 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:15.682972    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c3efb2-5f44b7cbcf-4zkz8" podStartSLOduration=2.682947296 podStartE2EDuration="2.682947296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:38:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:38:15.671322955 +0000 UTC m=+2114.818590197" watchObservedRunningTime="2026-10-30 13:38:15.682947296 +0000 UTC m=+2114.830214538"
Oct 30 13:38:21.243826 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:38:21.243778    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:38:24.187744 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:24.187702    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f659skml"]
Oct 30 13:38:24.192449 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:24.192421    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f659skml"
Oct 30 13:38:24.210591 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:24.210557    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f659skml"]
Oct 30 13:38:24.219095 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:24.219066    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e3eb98\"/\"agc-cred-94a5564a5433\""
Oct 30 13:38:24.219095 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:24.219066    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e3eb98\"/\"openshift-service-ca.crt\""
Oct 30 13:38:24.219324 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:24.219073    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e3eb98\"/\"payload-processing-external-model-e2e--2547b850-dockercfg-rj6kb\""
Oct 30 13:38:24.219324 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:24.219203    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e3eb98\"/\"kube-root-ca.crt\""
Oct 30 13:38:24.221854 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:24.221827    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e3eb98\"/\"payload-processing-external-model-plugins-e2e-extmod-b-e3eb98\""
Oct 30 13:38:24.221969 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:24.221889    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e3eb98\"/\"routing-overlay\""
Oct 30 13:38:24.281155 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:24.279701    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/21bb5813-fa7f-4d6b-9dd4-430cfc96e380-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-5f659skml\" (UID: \"21bb5813-fa7f-4d6b-9dd4-430cfc96e380\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f659skml"
Oct 30 13:38:24.281155 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:24.279927    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/21bb5813-fa7f-4d6b-9dd4-430cfc96e380-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-5f659skml\" (UID: \"21bb5813-fa7f-4d6b-9dd4-430cfc96e380\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f659skml"
Oct 30 13:38:24.281155 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:24.280028    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21bb5813-fa7f-4d6b-9dd4-430cfc96e380-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-5f659skml\" (UID: \"21bb5813-fa7f-4d6b-9dd4-430cfc96e380\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f659skml"
Oct 30 13:38:24.281155 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:24.280072    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bn8m\" (UniqueName: \"kubernetes.io/projected/21bb5813-fa7f-4d6b-9dd4-430cfc96e380-kube-api-access-9bn8m\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-5f659skml\" (UID: \"21bb5813-fa7f-4d6b-9dd4-430cfc96e380\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f659skml"
Oct 30 13:38:24.392099 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:24.381540    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/21bb5813-fa7f-4d6b-9dd4-430cfc96e380-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-5f659skml\" (UID: \"21bb5813-fa7f-4d6b-9dd4-430cfc96e380\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f659skml"
Oct 30 13:38:24.392099 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:24.381689    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21bb5813-fa7f-4d6b-9dd4-430cfc96e380-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-5f659skml\" (UID: \"21bb5813-fa7f-4d6b-9dd4-430cfc96e380\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f659skml"
Oct 30 13:38:24.392099 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:24.381739    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bn8m\" (UniqueName: \"kubernetes.io/projected/21bb5813-fa7f-4d6b-9dd4-430cfc96e380-kube-api-access-9bn8m\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-5f659skml\" (UID: \"21bb5813-fa7f-4d6b-9dd4-430cfc96e380\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f659skml"
Oct 30 13:38:24.392099 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:24.381828    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/21bb5813-fa7f-4d6b-9dd4-430cfc96e380-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-5f659skml\" (UID: \"21bb5813-fa7f-4d6b-9dd4-430cfc96e380\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f659skml"
Oct 30 13:38:24.392099 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:24.382855    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/21bb5813-fa7f-4d6b-9dd4-430cfc96e380-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-5f659skml\" (UID: \"21bb5813-fa7f-4d6b-9dd4-430cfc96e380\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f659skml"
Oct 30 13:38:24.392099 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:24.384490    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21bb5813-fa7f-4d6b-9dd4-430cfc96e380-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-5f659skml\" (UID: \"21bb5813-fa7f-4d6b-9dd4-430cfc96e380\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f659skml"
Oct 30 13:38:24.392099 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:24.387829    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/21bb5813-fa7f-4d6b-9dd4-430cfc96e380-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-5f659skml\" (UID: \"21bb5813-fa7f-4d6b-9dd4-430cfc96e380\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f659skml"
Oct 30 13:38:24.405356 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:24.405325    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bn8m\" (UniqueName: \"kubernetes.io/projected/21bb5813-fa7f-4d6b-9dd4-430cfc96e380-kube-api-access-9bn8m\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-5f659skml\" (UID: \"21bb5813-fa7f-4d6b-9dd4-430cfc96e380\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f659skml"
Oct 30 13:38:24.508086 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:24.507996    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f659skml"
Oct 30 13:38:24.825803 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:24.825768    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f659skml"]
Oct 30 13:38:24.825942 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:38:24.825810    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21bb5813_fa7f_4d6b_9dd4_430cfc96e380.slice/crio-1ee330cb583e54c65e3feaefc811fae43efd62c4d044ce0c45fd7dcb9823d107 WatchSource:0}: Error finding container 1ee330cb583e54c65e3feaefc811fae43efd62c4d044ce0c45fd7dcb9823d107: Status 404 returned error can't find the container with id 1ee330cb583e54c65e3feaefc811fae43efd62c4d044ce0c45fd7dcb9823d107
Oct 30 13:38:25.744513 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:25.744476    5093 generic.go:356] "Generic (PLEG): container finished" podID="b0d3d9d3-593a-478d-963b-d8b4b395a661" containerID="b3b733ee51a45db343a28e5099a1558c7507df79b6084d95825f8c769dc461d3" exitCode=0
Oct 30 13:38:25.745087 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:25.744570    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm" event={"ID":"b0d3d9d3-593a-478d-963b-d8b4b395a661","Type":"ContainerDied","Data":"b3b733ee51a45db343a28e5099a1558c7507df79b6084d95825f8c769dc461d3"}
Oct 30 13:38:25.745198 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:25.745167    5093 scope.go:122] "RemoveContainer" containerID="b3b733ee51a45db343a28e5099a1558c7507df79b6084d95825f8c769dc461d3"
Oct 30 13:38:25.747216 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:25.747184    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f659skml" event={"ID":"21bb5813-fa7f-4d6b-9dd4-430cfc96e380","Type":"ContainerStarted","Data":"a1baea85c1ce558d5789e747bc4bc0a954b275c16a8bb291b2c2a7e92404ed19"}
Oct 30 13:38:25.747362 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:25.747245    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f659skml" event={"ID":"21bb5813-fa7f-4d6b-9dd4-430cfc96e380","Type":"ContainerStarted","Data":"1ee330cb583e54c65e3feaefc811fae43efd62c4d044ce0c45fd7dcb9823d107"}
Oct 30 13:38:25.747567 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:25.747535    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:38:25.747640 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:25.747623    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:38:25.747693 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:25.747669    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:38:25.747964 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:25.747949    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f659skml"
Oct 30 13:38:25.854945 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:25.854792    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f659skml" podStartSLOduration=1.854769831 podStartE2EDuration="1.854769831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:38:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:38:25.847313467 +0000 UTC m=+2124.994580709" watchObservedRunningTime="2026-10-30 13:38:25.854769831 +0000 UTC m=+2125.002037074"
Oct 30 13:38:26.772953 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:26.772913    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm" event={"ID":"b0d3d9d3-593a-478d-963b-d8b4b395a661","Type":"ContainerStarted","Data":"4e34bdd9f7c4a57d1a0b007b163c1c302f5100fbb4884adc7c1c3b66a32afe10"}
Oct 30 13:38:27.415508 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:27.415468    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm"
Oct 30 13:38:27.415723 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:27.415678    5093 patch_prober.go:29] interesting pod/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 30 13:38:27.415800 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:27.415760    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm" podUID="b0d3d9d3-593a-478d-963b-d8b4b395a661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 30 13:38:27.666456 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:27.666374    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c3efb2-5f44b7cbcf-4zkz8"
Oct 30 13:38:28.416381 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:28.416343    5093 patch_prober.go:29] interesting pod/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 30 13:38:28.416795 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:28.416428    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm" podUID="b0d3d9d3-593a-478d-963b-d8b4b395a661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 30 13:38:29.416319 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:29.416281    5093 patch_prober.go:29] interesting pod/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 30 13:38:29.416510 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:29.416341    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm" podUID="b0d3d9d3-593a-478d-963b-d8b4b395a661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 30 13:38:30.416341 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:30.416303    5093 patch_prober.go:29] interesting pod/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 30 13:38:30.416531 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:30.416365    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm" podUID="b0d3d9d3-593a-478d-963b-d8b4b395a661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 30 13:38:31.415824 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:31.415781    5093 patch_prober.go:29] interesting pod/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 30 13:38:31.416033 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:31.415865    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm" podUID="b0d3d9d3-593a-478d-963b-d8b4b395a661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 30 13:38:32.421662 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:32.421618    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm"
Oct 30 13:38:32.422258 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:32.421990    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm"
Oct 30 13:38:32.422881 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:32.422857    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm"
Oct 30 13:38:36.777209 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:36.777174    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f659skml"
Oct 30 13:38:50.415994 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:50.415957    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c3efb2-6d4548b87-n444z"]
Oct 30 13:38:50.422340 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:50.422311    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c3efb2-6d4548b87-n444z"
Oct 30 13:38:50.440199 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:50.440163    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c3efb2-6d4548b87-n444z"]
Oct 30 13:38:50.447102 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:50.447067    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-c3efb2-serving-cert\""
Oct 30 13:38:50.530351 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:50.530312    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vv6v\" (UniqueName: \"kubernetes.io/projected/6b42343a-61e7-468a-aae9-701026a365bc-kube-api-access-7vv6v\") pod \"maas-api-e2e-extmod-a-c3efb2-6d4548b87-n444z\" (UID: \"6b42343a-61e7-468a-aae9-701026a365bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c3efb2-6d4548b87-n444z"
Oct 30 13:38:50.530351 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:50.530363    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b42343a-61e7-468a-aae9-701026a365bc-metrics-certs\") pod \"maas-api-e2e-extmod-a-c3efb2-6d4548b87-n444z\" (UID: \"6b42343a-61e7-468a-aae9-701026a365bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c3efb2-6d4548b87-n444z"
Oct 30 13:38:50.530580 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:50.530538    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b42343a-61e7-468a-aae9-701026a365bc-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c3efb2-6d4548b87-n444z\" (UID: \"6b42343a-61e7-468a-aae9-701026a365bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c3efb2-6d4548b87-n444z"
Oct 30 13:38:50.530699 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:50.530686    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b42343a-61e7-468a-aae9-701026a365bc-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c3efb2-6d4548b87-n444z\" (UID: \"6b42343a-61e7-468a-aae9-701026a365bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c3efb2-6d4548b87-n444z"
Oct 30 13:38:50.631941 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:50.631901    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b42343a-61e7-468a-aae9-701026a365bc-metrics-certs\") pod \"maas-api-e2e-extmod-a-c3efb2-6d4548b87-n444z\" (UID: \"6b42343a-61e7-468a-aae9-701026a365bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c3efb2-6d4548b87-n444z"
Oct 30 13:38:50.632155 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:50.632041    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b42343a-61e7-468a-aae9-701026a365bc-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c3efb2-6d4548b87-n444z\" (UID: \"6b42343a-61e7-468a-aae9-701026a365bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c3efb2-6d4548b87-n444z"
Oct 30 13:38:50.632281 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:50.632263    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b42343a-61e7-468a-aae9-701026a365bc-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c3efb2-6d4548b87-n444z\" (UID: \"6b42343a-61e7-468a-aae9-701026a365bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c3efb2-6d4548b87-n444z"
Oct 30 13:38:50.632406 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:50.632381    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vv6v\" (UniqueName: \"kubernetes.io/projected/6b42343a-61e7-468a-aae9-701026a365bc-kube-api-access-7vv6v\") pod \"maas-api-e2e-extmod-a-c3efb2-6d4548b87-n444z\" (UID: \"6b42343a-61e7-468a-aae9-701026a365bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c3efb2-6d4548b87-n444z"
Oct 30 13:38:50.632927 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:50.632901    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b42343a-61e7-468a-aae9-701026a365bc-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c3efb2-6d4548b87-n444z\" (UID: \"6b42343a-61e7-468a-aae9-701026a365bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c3efb2-6d4548b87-n444z"
Oct 30 13:38:50.635222 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:50.635197    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b42343a-61e7-468a-aae9-701026a365bc-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c3efb2-6d4548b87-n444z\" (UID: \"6b42343a-61e7-468a-aae9-701026a365bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c3efb2-6d4548b87-n444z"
Oct 30 13:38:50.635776 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:50.635732    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b42343a-61e7-468a-aae9-701026a365bc-metrics-certs\") pod \"maas-api-e2e-extmod-a-c3efb2-6d4548b87-n444z\" (UID: \"6b42343a-61e7-468a-aae9-701026a365bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c3efb2-6d4548b87-n444z"
Oct 30 13:38:50.661714 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:50.661686    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vv6v\" (UniqueName: \"kubernetes.io/projected/6b42343a-61e7-468a-aae9-701026a365bc-kube-api-access-7vv6v\") pod \"maas-api-e2e-extmod-a-c3efb2-6d4548b87-n444z\" (UID: \"6b42343a-61e7-468a-aae9-701026a365bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c3efb2-6d4548b87-n444z"
Oct 30 13:38:50.741154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:50.740612    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c3efb2-6d4548b87-n444z"
Oct 30 13:38:51.170113 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:38:51.170011    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:38:51.180079 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:51.180052    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c3efb2-6d4548b87-n444z"]
Oct 30 13:38:51.180656 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:38:51.180633    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b42343a_61e7_468a_aae9_701026a365bc.slice/crio-25814f9722072da28fcd46f33c879cd4931b2134b56084edb640be0fa613efee WatchSource:0}: Error finding container 25814f9722072da28fcd46f33c879cd4931b2134b56084edb640be0fa613efee: Status 404 returned error can't find the container with id 25814f9722072da28fcd46f33c879cd4931b2134b56084edb640be0fa613efee
Oct 30 13:38:52.030658 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:52.030621    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c3efb2-6d4548b87-n444z" event={"ID":"6b42343a-61e7-468a-aae9-701026a365bc","Type":"ContainerStarted","Data":"63843bd2872be329fb9394fdc18a1b39682ce01d73eea1d94f325d0870b96669"}
Oct 30 13:38:52.031061 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:52.030678    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c3efb2-6d4548b87-n444z" event={"ID":"6b42343a-61e7-468a-aae9-701026a365bc","Type":"ContainerStarted","Data":"25814f9722072da28fcd46f33c879cd4931b2134b56084edb640be0fa613efee"}
Oct 30 13:38:52.031061 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:52.030701    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c3efb2-6d4548b87-n444z"
Oct 30 13:38:52.075400 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:52.075344    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c3efb2-6d4548b87-n444z" podStartSLOduration=1.60022322 podStartE2EDuration="2.075327632s" totalImagesPullingTime="475.104412ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:38:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:38:52.072769715 +0000 UTC m=+2151.220036969" watchObservedRunningTime="2026-10-30 13:38:52.075327632 +0000 UTC m=+2151.222594874"
Oct 30 13:38:58.045325 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:38:58.045290    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c3efb2-6d4548b87-n444z"
Oct 30 13:39:01.990177 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:39:01.990120    5093 scope.go:122] "RemoveContainer" containerID="50e24476000696345ec9607a66e83c774dfae3403d41ba7f6d38d0cb96eecbfa"
Oct 30 13:39:21.169456 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:39:21.169405    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:39:43.346657 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:39:43.346550    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-75c46dbdp"]
Oct 30 13:39:43.352484 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:39:43.352457    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-75c46dbdp"
Oct 30 13:39:43.373713 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:39:43.373681    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-75c46dbdp"]
Oct 30 13:39:43.381715 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:39:43.381682    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-844856\"/\"payload-processing-external-model-e2e--cbdef895-dockercfg-rrsgw\""
Oct 30 13:39:43.381893 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:39:43.381792    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-844856\"/\"kube-root-ca.crt\""
Oct 30 13:39:43.382379 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:39:43.382353    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-844856\"/\"openshift-service-ca.crt\""
Oct 30 13:39:43.382527 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:39:43.382454    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-844856\"/\"routing-overlay\""
Oct 30 13:39:43.382527 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:39:43.382357    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-844856\"/\"agc-cred-62b7f7e20837\""
Oct 30 13:39:43.382527 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:39:43.382517    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-844856\"/\"agc-cred-a2fb6db3f151\""
Oct 30 13:39:43.430763 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:39:43.430708    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjzl2\" (UniqueName: \"kubernetes.io/projected/61f081bf-937f-454f-a1a0-218174acf4c5-kube-api-access-wjzl2\") pod \"payload-processing-external-model-e2e-extmod-b-844856-75c46dbdp\" (UID: \"61f081bf-937f-454f-a1a0-218174acf4c5\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-75c46dbdp"
Oct 30 13:39:43.430965 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:39:43.430869    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61f081bf-937f-454f-a1a0-218174acf4c5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-844856-75c46dbdp\" (UID: \"61f081bf-937f-454f-a1a0-218174acf4c5\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-75c46dbdp"
Oct 30 13:39:43.430965 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:39:43.430907    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/61f081bf-937f-454f-a1a0-218174acf4c5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-844856-75c46dbdp\" (UID: \"61f081bf-937f-454f-a1a0-218174acf4c5\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-75c46dbdp"
Oct 30 13:39:43.431067 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:39:43.431004    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/61f081bf-937f-454f-a1a0-218174acf4c5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-844856-75c46dbdp\" (UID: \"61f081bf-937f-454f-a1a0-218174acf4c5\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-75c46dbdp"
Oct 30 13:39:43.469020 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:39:43.468978    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-844856\"/\"payload-processing-external-model-plugins-e2e-extmod-b-844856\""
Oct 30 13:39:43.469258 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:39:43.469173    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-844856\"/\"agc-cred-006c41e31efa\""
Oct 30 13:39:43.532125 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:39:43.532090    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjzl2\" (UniqueName: \"kubernetes.io/projected/61f081bf-937f-454f-a1a0-218174acf4c5-kube-api-access-wjzl2\") pod \"payload-processing-external-model-e2e-extmod-b-844856-75c46dbdp\" (UID: \"61f081bf-937f-454f-a1a0-218174acf4c5\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-75c46dbdp"
Oct 30 13:39:43.532332 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:39:43.532236    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61f081bf-937f-454f-a1a0-218174acf4c5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-844856-75c46dbdp\" (UID: \"61f081bf-937f-454f-a1a0-218174acf4c5\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-75c46dbdp"
Oct 30 13:39:43.532332 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:39:43.532275    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/61f081bf-937f-454f-a1a0-218174acf4c5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-844856-75c46dbdp\" (UID: \"61f081bf-937f-454f-a1a0-218174acf4c5\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-75c46dbdp"
Oct 30 13:39:43.532455 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:39:43.532409    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/61f081bf-937f-454f-a1a0-218174acf4c5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-844856-75c46dbdp\" (UID: \"61f081bf-937f-454f-a1a0-218174acf4c5\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-75c46dbdp"
Oct 30 13:39:43.533212 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:39:43.533186    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61f081bf-937f-454f-a1a0-218174acf4c5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-844856-75c46dbdp\" (UID: \"61f081bf-937f-454f-a1a0-218174acf4c5\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-75c46dbdp"
Oct 30 13:39:43.533324 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:39:43.533242    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/61f081bf-937f-454f-a1a0-218174acf4c5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-844856-75c46dbdp\" (UID: \"61f081bf-937f-454f-a1a0-218174acf4c5\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-75c46dbdp"
Oct 30 13:39:43.535022 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:39:43.534996    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/61f081bf-937f-454f-a1a0-218174acf4c5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-844856-75c46dbdp\" (UID: \"61f081bf-937f-454f-a1a0-218174acf4c5\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-75c46dbdp"
Oct 30 13:39:43.553586 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:39:43.553117    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjzl2\" (UniqueName: \"kubernetes.io/projected/61f081bf-937f-454f-a1a0-218174acf4c5-kube-api-access-wjzl2\") pod \"payload-processing-external-model-e2e-extmod-b-844856-75c46dbdp\" (UID: \"61f081bf-937f-454f-a1a0-218174acf4c5\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-75c46dbdp"
Oct 30 13:39:43.670557 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:39:43.670470    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-75c46dbdp"
Oct 30 13:39:44.051719 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:39:44.051681    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-75c46dbdp"]
Oct 30 13:39:44.492698 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:39:44.492653    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-75c46dbdp" event={"ID":"61f081bf-937f-454f-a1a0-218174acf4c5","Type":"ContainerStarted","Data":"5187ea5e5c6b439a72490884ab2a0f30d48fa7b517bca8fe312ca590b12a5b59"}
Oct 30 13:39:44.493170 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:39:44.492726    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-75c46dbdp" event={"ID":"61f081bf-937f-454f-a1a0-218174acf4c5","Type":"ContainerStarted","Data":"e1b947968ff3a9e8907a64ccca939871421a3be37568598573c2222b6cd6f805"}
Oct 30 13:39:44.493170 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:39:44.492755    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-75c46dbdp"
Oct 30 13:39:44.537253 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:39:44.537184    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-75c46dbdp" podStartSLOduration=1.5371702059999999 podStartE2EDuration="1.537170206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:39:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:39:44.53271064 +0000 UTC m=+2203.679977879" watchObservedRunningTime="2026-10-30 13:39:44.537170206 +0000 UTC m=+2203.684437473"
Oct 30 13:39:46.625174 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:39:46.625122    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a75d02-7f44cf6f8c-86ss9"]
Oct 30 13:39:46.631195 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:39:46.631170    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a75d02-7f44cf6f8c-86ss9"
Oct 30 13:39:46.650696 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:39:46.650662    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a75d02-7f44cf6f8c-86ss9"]
Oct 30 13:39:46.659447 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:39:46.658850    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-a75d02\""
Oct 30 13:39:46.661069 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:39:46.660226    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-a75d02-dockercfg-svs5v\""
Oct 30 13:39:46.691168 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:39:46.689780    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj8bz\" (UniqueName: \"kubernetes.io/projected/fa8178a9-663a-44bd-ba75-c9c521f3045e-kube-api-access-wj8bz\") pod \"payload-pre-processing-e2e-extmod-b-a75d02-7f44cf6f8c-86ss9\" (UID: \"fa8178a9-663a-44bd-ba75-c9c521f3045e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a75d02-7f44cf6f8c-86ss9"
Oct 30 13:39:46.691168 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:39:46.690042    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa8178a9-663a-44bd-ba75-c9c521f3045e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-a75d02-7f44cf6f8c-86ss9\" (UID: \"fa8178a9-663a-44bd-ba75-c9c521f3045e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a75d02-7f44cf6f8c-86ss9"
Oct 30 13:39:46.807157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:39:46.801635    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj8bz\" (UniqueName: \"kubernetes.io/projected/fa8178a9-663a-44bd-ba75-c9c521f3045e-kube-api-access-wj8bz\") pod \"payload-pre-processing-e2e-extmod-b-a75d02-7f44cf6f8c-86ss9\" (UID: \"fa8178a9-663a-44bd-ba75-c9c521f3045e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a75d02-7f44cf6f8c-86ss9"
Oct 30 13:39:46.807157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:39:46.801831    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa8178a9-663a-44bd-ba75-c9c521f3045e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-a75d02-7f44cf6f8c-86ss9\" (UID: \"fa8178a9-663a-44bd-ba75-c9c521f3045e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a75d02-7f44cf6f8c-86ss9"
Oct 30 13:39:46.807157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:39:46.802988    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa8178a9-663a-44bd-ba75-c9c521f3045e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-a75d02-7f44cf6f8c-86ss9\" (UID: \"fa8178a9-663a-44bd-ba75-c9c521f3045e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a75d02-7f44cf6f8c-86ss9"
Oct 30 13:39:46.844009 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:39:46.833857    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj8bz\" (UniqueName: \"kubernetes.io/projected/fa8178a9-663a-44bd-ba75-c9c521f3045e-kube-api-access-wj8bz\") pod \"payload-pre-processing-e2e-extmod-b-a75d02-7f44cf6f8c-86ss9\" (UID: \"fa8178a9-663a-44bd-ba75-c9c521f3045e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a75d02-7f44cf6f8c-86ss9"
Oct 30 13:39:46.947938 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:39:46.947899    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a75d02-7f44cf6f8c-86ss9"
Oct 30 13:39:47.256383 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:39:47.256201    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa8178a9_663a_44bd_ba75_c9c521f3045e.slice/crio-1ea57ebec85c683a089ac035f6fa11c584b4619801d52f9cef7bb3f19eb3c79b WatchSource:0}: Error finding container 1ea57ebec85c683a089ac035f6fa11c584b4619801d52f9cef7bb3f19eb3c79b: Status 404 returned error can't find the container with id 1ea57ebec85c683a089ac035f6fa11c584b4619801d52f9cef7bb3f19eb3c79b
Oct 30 13:39:47.261942 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:39:47.261909    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a75d02-7f44cf6f8c-86ss9"]
Oct 30 13:39:47.522642 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:39:47.522534    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a75d02-7f44cf6f8c-86ss9" event={"ID":"fa8178a9-663a-44bd-ba75-c9c521f3045e","Type":"ContainerStarted","Data":"0865d25b7d4e55f280de43c5fc67437c9d7b9a54a94160e4deb9ff0d6cfc118f"}
Oct 30 13:39:47.522642 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:39:47.522610    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a75d02-7f44cf6f8c-86ss9" event={"ID":"fa8178a9-663a-44bd-ba75-c9c521f3045e","Type":"ContainerStarted","Data":"1ea57ebec85c683a089ac035f6fa11c584b4619801d52f9cef7bb3f19eb3c79b"}
Oct 30 13:39:47.522642 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:39:47.522644    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a75d02-7f44cf6f8c-86ss9"
Oct 30 13:39:47.564101 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:39:47.564032    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a75d02-7f44cf6f8c-86ss9" podStartSLOduration=1.5640132530000002 podStartE2EDuration="1.564013253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:39:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:39:47.563076726 +0000 UTC m=+2206.710343968" watchObservedRunningTime="2026-10-30 13:39:47.564013253 +0000 UTC m=+2206.711280494"
Oct 30 13:39:51.169949 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:39:51.169906    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:39:55.428625 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:39:55.428580    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f659skml"]
Oct 30 13:39:55.504407 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:39:55.504375    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-75c46dbdp"
Oct 30 13:39:58.536063 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:39:58.536028    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a75d02-7f44cf6f8c-86ss9"
Oct 30 13:40:00.445365 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:00.437274    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f659skml" podUID="21bb5813-fa7f-4d6b-9dd4-430cfc96e380" containerName="payload-processing" containerID="cri-o://a1baea85c1ce558d5789e747bc4bc0a954b275c16a8bb291b2c2a7e92404ed19" gracePeriod=25
Oct 30 13:40:00.663633 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:00.663516    5093 generic.go:356] "Generic (PLEG): container finished" podID="21bb5813-fa7f-4d6b-9dd4-430cfc96e380" containerID="a1baea85c1ce558d5789e747bc4bc0a954b275c16a8bb291b2c2a7e92404ed19" exitCode=0
Oct 30 13:40:00.663633 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:00.663585    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f659skml" event={"ID":"21bb5813-fa7f-4d6b-9dd4-430cfc96e380","Type":"ContainerDied","Data":"a1baea85c1ce558d5789e747bc4bc0a954b275c16a8bb291b2c2a7e92404ed19"}
Oct 30 13:40:00.760193 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:00.760160    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f659skml"
Oct 30 13:40:00.815694 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:00.815654    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21bb5813-fa7f-4d6b-9dd4-430cfc96e380-provider-credentials\" (UniqueName: \"kubernetes.io/projected/21bb5813-fa7f-4d6b-9dd4-430cfc96e380-provider-credentials\") pod \"21bb5813-fa7f-4d6b-9dd4-430cfc96e380\" (UID: \"21bb5813-fa7f-4d6b-9dd4-430cfc96e380\") "
Oct 30 13:40:00.815904 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:00.815743    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21bb5813-fa7f-4d6b-9dd4-430cfc96e380-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21bb5813-fa7f-4d6b-9dd4-430cfc96e380-plugins-config-volume\") pod \"21bb5813-fa7f-4d6b-9dd4-430cfc96e380\" (UID: \"21bb5813-fa7f-4d6b-9dd4-430cfc96e380\") "
Oct 30 13:40:00.816308 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:00.816030    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21bb5813-fa7f-4d6b-9dd4-430cfc96e380-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/21bb5813-fa7f-4d6b-9dd4-430cfc96e380-routing-overlay-volume\") pod \"21bb5813-fa7f-4d6b-9dd4-430cfc96e380\" (UID: \"21bb5813-fa7f-4d6b-9dd4-430cfc96e380\") "
Oct 30 13:40:00.816308 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:00.816108    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21bb5813-fa7f-4d6b-9dd4-430cfc96e380-kube-api-access-9bn8m\" (UniqueName: \"kubernetes.io/projected/21bb5813-fa7f-4d6b-9dd4-430cfc96e380-kube-api-access-9bn8m\") pod \"21bb5813-fa7f-4d6b-9dd4-430cfc96e380\" (UID: \"21bb5813-fa7f-4d6b-9dd4-430cfc96e380\") "
Oct 30 13:40:00.817009 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:00.816727    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21bb5813-fa7f-4d6b-9dd4-430cfc96e380-plugins-config-volume" pod "21bb5813-fa7f-4d6b-9dd4-430cfc96e380" (UID: "21bb5813-fa7f-4d6b-9dd4-430cfc96e380"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:00.817355 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:00.817301    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21bb5813-fa7f-4d6b-9dd4-430cfc96e380-routing-overlay-volume" pod "21bb5813-fa7f-4d6b-9dd4-430cfc96e380" (UID: "21bb5813-fa7f-4d6b-9dd4-430cfc96e380"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:00.822742 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:00.822698    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21bb5813-fa7f-4d6b-9dd4-430cfc96e380-provider-credentials" pod "21bb5813-fa7f-4d6b-9dd4-430cfc96e380" (UID: "21bb5813-fa7f-4d6b-9dd4-430cfc96e380"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:00.822857 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:00.822733    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21bb5813-fa7f-4d6b-9dd4-430cfc96e380-kube-api-access-9bn8m" pod "21bb5813-fa7f-4d6b-9dd4-430cfc96e380" (UID: "21bb5813-fa7f-4d6b-9dd4-430cfc96e380"). InnerVolumeSpecName "kube-api-access-9bn8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:00.918428 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:00.918395    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bn8m\" (UniqueName: \"kubernetes.io/projected/21bb5813-fa7f-4d6b-9dd4-430cfc96e380-kube-api-access-9bn8m\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:40:00.918428 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:00.918424    5093 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/21bb5813-fa7f-4d6b-9dd4-430cfc96e380-provider-credentials\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:40:00.918645 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:00.918436    5093 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21bb5813-fa7f-4d6b-9dd4-430cfc96e380-plugins-config-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:40:00.918645 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:00.918447    5093 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/21bb5813-fa7f-4d6b-9dd4-430cfc96e380-routing-overlay-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:40:01.673839 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:01.673809    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f659skml"
Oct 30 13:40:01.673839 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:01.673829    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f659skml" event={"ID":"21bb5813-fa7f-4d6b-9dd4-430cfc96e380","Type":"ContainerDied","Data":"1ee330cb583e54c65e3feaefc811fae43efd62c4d044ce0c45fd7dcb9823d107"}
Oct 30 13:40:01.674416 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:01.673904    5093 scope.go:122] "RemoveContainer" containerID="a1baea85c1ce558d5789e747bc4bc0a954b275c16a8bb291b2c2a7e92404ed19"
Oct 30 13:40:01.737588 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:01.737548    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f659skml"]
Oct 30 13:40:01.749076 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:01.749040    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-5f659skml"]
Oct 30 13:40:03.000606 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:03.000572    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21bb5813-fa7f-4d6b-9dd4-430cfc96e380" path="/var/lib/kubelet/pods/21bb5813-fa7f-4d6b-9dd4-430cfc96e380/volumes"
Oct 30 13:40:21.169478 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:40:21.169433    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:40:25.350778 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:25.350739    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a75d02-79d466cff8-64rb8"]
Oct 30 13:40:25.351545 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:25.351528    5093 cpu_manager.go:414] "containerMap: removing container" podUID="21bb5813-fa7f-4d6b-9dd4-430cfc96e380" containerName="payload-processing"
Oct 30 13:40:25.351593 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:25.351547    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21bb5813-fa7f-4d6b-9dd4-430cfc96e380" containerName="payload-processing"
Oct 30 13:40:25.351689 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:25.351680    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="21bb5813-fa7f-4d6b-9dd4-430cfc96e380" containerName="payload-processing"
Oct 30 13:40:25.355452 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:25.355431    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a75d02-79d466cff8-64rb8"
Oct 30 13:40:25.374156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:25.374112    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a75d02-79d466cff8-64rb8"]
Oct 30 13:40:25.388560 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:25.388521    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-a75d02-serving-cert\""
Oct 30 13:40:25.478569 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:25.478532    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dba480c3-b59a-4acb-a634-6aa1f50de647-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-a75d02-79d466cff8-64rb8\" (UID: \"dba480c3-b59a-4acb-a634-6aa1f50de647\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a75d02-79d466cff8-64rb8"
Oct 30 13:40:25.478745 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:25.478648    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dba480c3-b59a-4acb-a634-6aa1f50de647-maas-api-tls\") pod \"maas-api-e2e-extmod-b-a75d02-79d466cff8-64rb8\" (UID: \"dba480c3-b59a-4acb-a634-6aa1f50de647\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a75d02-79d466cff8-64rb8"
Oct 30 13:40:25.478745 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:25.478681    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dba480c3-b59a-4acb-a634-6aa1f50de647-metrics-certs\") pod \"maas-api-e2e-extmod-b-a75d02-79d466cff8-64rb8\" (UID: \"dba480c3-b59a-4acb-a634-6aa1f50de647\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a75d02-79d466cff8-64rb8"
Oct 30 13:40:25.478840 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:25.478825    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmdmj\" (UniqueName: \"kubernetes.io/projected/dba480c3-b59a-4acb-a634-6aa1f50de647-kube-api-access-tmdmj\") pod \"maas-api-e2e-extmod-b-a75d02-79d466cff8-64rb8\" (UID: \"dba480c3-b59a-4acb-a634-6aa1f50de647\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a75d02-79d466cff8-64rb8"
Oct 30 13:40:25.580163 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:25.580111    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dba480c3-b59a-4acb-a634-6aa1f50de647-maas-api-tls\") pod \"maas-api-e2e-extmod-b-a75d02-79d466cff8-64rb8\" (UID: \"dba480c3-b59a-4acb-a634-6aa1f50de647\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a75d02-79d466cff8-64rb8"
Oct 30 13:40:25.580460 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:25.580431    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dba480c3-b59a-4acb-a634-6aa1f50de647-metrics-certs\") pod \"maas-api-e2e-extmod-b-a75d02-79d466cff8-64rb8\" (UID: \"dba480c3-b59a-4acb-a634-6aa1f50de647\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a75d02-79d466cff8-64rb8"
Oct 30 13:40:25.580652 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:25.580629    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmdmj\" (UniqueName: \"kubernetes.io/projected/dba480c3-b59a-4acb-a634-6aa1f50de647-kube-api-access-tmdmj\") pod \"maas-api-e2e-extmod-b-a75d02-79d466cff8-64rb8\" (UID: \"dba480c3-b59a-4acb-a634-6aa1f50de647\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a75d02-79d466cff8-64rb8"
Oct 30 13:40:25.580841 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:25.580813    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dba480c3-b59a-4acb-a634-6aa1f50de647-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-a75d02-79d466cff8-64rb8\" (UID: \"dba480c3-b59a-4acb-a634-6aa1f50de647\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a75d02-79d466cff8-64rb8"
Oct 30 13:40:25.582003 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:25.581962    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dba480c3-b59a-4acb-a634-6aa1f50de647-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-a75d02-79d466cff8-64rb8\" (UID: \"dba480c3-b59a-4acb-a634-6aa1f50de647\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a75d02-79d466cff8-64rb8"
Oct 30 13:40:25.583867 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:25.583838    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dba480c3-b59a-4acb-a634-6aa1f50de647-metrics-certs\") pod \"maas-api-e2e-extmod-b-a75d02-79d466cff8-64rb8\" (UID: \"dba480c3-b59a-4acb-a634-6aa1f50de647\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a75d02-79d466cff8-64rb8"
Oct 30 13:40:25.589893 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:25.588950    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dba480c3-b59a-4acb-a634-6aa1f50de647-maas-api-tls\") pod \"maas-api-e2e-extmod-b-a75d02-79d466cff8-64rb8\" (UID: \"dba480c3-b59a-4acb-a634-6aa1f50de647\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a75d02-79d466cff8-64rb8"
Oct 30 13:40:25.602855 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:25.602775    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmdmj\" (UniqueName: \"kubernetes.io/projected/dba480c3-b59a-4acb-a634-6aa1f50de647-kube-api-access-tmdmj\") pod \"maas-api-e2e-extmod-b-a75d02-79d466cff8-64rb8\" (UID: \"dba480c3-b59a-4acb-a634-6aa1f50de647\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a75d02-79d466cff8-64rb8"
Oct 30 13:40:25.671598 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:25.671563    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a75d02-79d466cff8-64rb8"
Oct 30 13:40:25.981793 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:25.981764    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a75d02-79d466cff8-64rb8"]
Oct 30 13:40:25.982772 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:40:25.982743    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddba480c3_b59a_4acb_a634_6aa1f50de647.slice/crio-5cd59ff5e034562f2dcd08cea13e5e346fd739b77dd7d0ef5061d0bd05990799 WatchSource:0}: Error finding container 5cd59ff5e034562f2dcd08cea13e5e346fd739b77dd7d0ef5061d0bd05990799: Status 404 returned error can't find the container with id 5cd59ff5e034562f2dcd08cea13e5e346fd739b77dd7d0ef5061d0bd05990799
Oct 30 13:40:26.892026 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:26.891977    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a75d02-79d466cff8-64rb8" event={"ID":"dba480c3-b59a-4acb-a634-6aa1f50de647","Type":"ContainerStarted","Data":"04dbb041b73d8d2878a21197c8ccae16f0c2bb3cb1a24355b453b1e7ed8b84f4"}
Oct 30 13:40:26.892704 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:26.892119    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a75d02-79d466cff8-64rb8" event={"ID":"dba480c3-b59a-4acb-a634-6aa1f50de647","Type":"ContainerStarted","Data":"5cd59ff5e034562f2dcd08cea13e5e346fd739b77dd7d0ef5061d0bd05990799"}
Oct 30 13:40:26.893009 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:26.892986    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a75d02-79d466cff8-64rb8"
Oct 30 13:40:26.936839 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:26.936768    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a75d02-79d466cff8-64rb8" podStartSLOduration=1.598989005 podStartE2EDuration="1.936753439s" totalImagesPullingTime="337.764434ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:40:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:40:26.932947688 +0000 UTC m=+2246.080214931" watchObservedRunningTime="2026-10-30 13:40:26.936753439 +0000 UTC m=+2246.084020679"
Oct 30 13:40:32.914635 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:32.914602    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a75d02-79d466cff8-64rb8"
Oct 30 13:40:46.542440 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:46.542404    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775d48thg"]
Oct 30 13:40:46.547551 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:46.547524    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775d48thg"
Oct 30 13:40:46.566251 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:46.566222    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775d48thg"]
Oct 30 13:40:46.573600 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:46.573194    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"agc-cred-587ab343f202\""
Oct 30 13:40:46.573600 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:46.573368    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"payload-processing-external-model-e2e--1c277df7-dockercfg-wk62z\""
Oct 30 13:40:46.573600 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:46.573485    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"routing-overlay\""
Oct 30 13:40:46.573600 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:46.573544    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"payload-processing-external-model-plugins-e2e-extmod-a-74589c\""
Oct 30 13:40:46.576892 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:46.576861    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"kube-root-ca.crt\""
Oct 30 13:40:46.576892 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:46.576886    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"openshift-service-ca.crt\""
Oct 30 13:40:46.625686 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:46.625648    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54gp4\" (UniqueName: \"kubernetes.io/projected/44b2314d-1921-47ee-9b8e-65548faacc0f-kube-api-access-54gp4\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-775d48thg\" (UID: \"44b2314d-1921-47ee-9b8e-65548faacc0f\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775d48thg"
Oct 30 13:40:46.625891 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:46.625727    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/44b2314d-1921-47ee-9b8e-65548faacc0f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-775d48thg\" (UID: \"44b2314d-1921-47ee-9b8e-65548faacc0f\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775d48thg"
Oct 30 13:40:46.625891 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:46.625859    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44b2314d-1921-47ee-9b8e-65548faacc0f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-775d48thg\" (UID: \"44b2314d-1921-47ee-9b8e-65548faacc0f\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775d48thg"
Oct 30 13:40:46.626011 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:46.626000    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/44b2314d-1921-47ee-9b8e-65548faacc0f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-775d48thg\" (UID: \"44b2314d-1921-47ee-9b8e-65548faacc0f\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775d48thg"
Oct 30 13:40:46.740150 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:46.738545    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/44b2314d-1921-47ee-9b8e-65548faacc0f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-775d48thg\" (UID: \"44b2314d-1921-47ee-9b8e-65548faacc0f\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775d48thg"
Oct 30 13:40:46.740150 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:46.738656    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44b2314d-1921-47ee-9b8e-65548faacc0f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-775d48thg\" (UID: \"44b2314d-1921-47ee-9b8e-65548faacc0f\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775d48thg"
Oct 30 13:40:46.740150 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:46.738781    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/44b2314d-1921-47ee-9b8e-65548faacc0f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-775d48thg\" (UID: \"44b2314d-1921-47ee-9b8e-65548faacc0f\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775d48thg"
Oct 30 13:40:46.740150 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:46.738934    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54gp4\" (UniqueName: \"kubernetes.io/projected/44b2314d-1921-47ee-9b8e-65548faacc0f-kube-api-access-54gp4\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-775d48thg\" (UID: \"44b2314d-1921-47ee-9b8e-65548faacc0f\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775d48thg"
Oct 30 13:40:46.757149 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:46.740616    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44b2314d-1921-47ee-9b8e-65548faacc0f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-775d48thg\" (UID: \"44b2314d-1921-47ee-9b8e-65548faacc0f\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775d48thg"
Oct 30 13:40:46.757149 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:46.741230    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/44b2314d-1921-47ee-9b8e-65548faacc0f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-775d48thg\" (UID: \"44b2314d-1921-47ee-9b8e-65548faacc0f\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775d48thg"
Oct 30 13:40:46.767397 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:46.762153    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/44b2314d-1921-47ee-9b8e-65548faacc0f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-775d48thg\" (UID: \"44b2314d-1921-47ee-9b8e-65548faacc0f\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775d48thg"
Oct 30 13:40:46.767397 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:46.762304    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54gp4\" (UniqueName: \"kubernetes.io/projected/44b2314d-1921-47ee-9b8e-65548faacc0f-kube-api-access-54gp4\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-775d48thg\" (UID: \"44b2314d-1921-47ee-9b8e-65548faacc0f\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775d48thg"
Oct 30 13:40:46.865030 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:46.864946    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775d48thg"
Oct 30 13:40:47.193206 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:47.193172    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775d48thg"]
Oct 30 13:40:47.194654 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:40:47.194624    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44b2314d_1921_47ee_9b8e_65548faacc0f.slice/crio-22e7f504adc1ce34a6209bba807251eaded5b867f296ea11159bceac1cced98c WatchSource:0}: Error finding container 22e7f504adc1ce34a6209bba807251eaded5b867f296ea11159bceac1cced98c: Status 404 returned error can't find the container with id 22e7f504adc1ce34a6209bba807251eaded5b867f296ea11159bceac1cced98c
Oct 30 13:40:48.169154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:48.168709    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775d48thg" event={"ID":"44b2314d-1921-47ee-9b8e-65548faacc0f","Type":"ContainerStarted","Data":"617a18df4e378bf68ce1380f23cbcc106e68a5bd8f7e75f9b60b846385aa435a"}
Oct 30 13:40:48.169154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:48.169018    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775d48thg" event={"ID":"44b2314d-1921-47ee-9b8e-65548faacc0f","Type":"ContainerStarted","Data":"22e7f504adc1ce34a6209bba807251eaded5b867f296ea11159bceac1cced98c"}
Oct 30 13:40:48.191175 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:48.173228    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775d48thg"
Oct 30 13:40:48.220694 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:48.219838    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775d48thg" podStartSLOduration=2.219815833 podStartE2EDuration="2.219815833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:40:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:40:48.212532185 +0000 UTC m=+2267.359799429" watchObservedRunningTime="2026-10-30 13:40:48.219815833 +0000 UTC m=+2267.367083079"
Oct 30 13:40:48.345185 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:40:48.336849    5093 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-74589c/agc-cred-587ab343f202: secret "agc-cred-587ab343f202" not found
Oct 30 13:40:48.345185 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:40:48.336881    5093 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775d48thg: secret "agc-cred-587ab343f202" not found
Oct 30 13:40:48.345185 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:40:48.336942    5093 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44b2314d-1921-47ee-9b8e-65548faacc0f-provider-credentials podName:44b2314d-1921-47ee-9b8e-65548faacc0f nodeName:}" failed. No retries permitted until 2026-10-30 13:40:48.836922081 +0000 UTC m=+2267.984189302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/44b2314d-1921-47ee-9b8e-65548faacc0f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-74589c-775d48thg" (UID: "44b2314d-1921-47ee-9b8e-65548faacc0f") : secret "agc-cred-587ab343f202" not found
Oct 30 13:40:48.843123 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:40:48.843084    5093 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-74589c/agc-cred-587ab343f202: secret "agc-cred-587ab343f202" not found
Oct 30 13:40:48.843123 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:40:48.843117    5093 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775d48thg: secret "agc-cred-587ab343f202" not found
Oct 30 13:40:48.843371 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:40:48.843217    5093 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44b2314d-1921-47ee-9b8e-65548faacc0f-provider-credentials podName:44b2314d-1921-47ee-9b8e-65548faacc0f nodeName:}" failed. No retries permitted until 2026-10-30 13:40:49.84319433 +0000 UTC m=+2268.990461562 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/44b2314d-1921-47ee-9b8e-65548faacc0f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-74589c-775d48thg" (UID: "44b2314d-1921-47ee-9b8e-65548faacc0f") : secret "agc-cred-587ab343f202" not found
Oct 30 13:40:49.857547 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:40:49.857510    5093 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-74589c/agc-cred-587ab343f202: secret "agc-cred-587ab343f202" not found
Oct 30 13:40:49.857547 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:40:49.857541    5093 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775d48thg: secret "agc-cred-587ab343f202" not found
Oct 30 13:40:49.858091 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:40:49.857614    5093 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44b2314d-1921-47ee-9b8e-65548faacc0f-provider-credentials podName:44b2314d-1921-47ee-9b8e-65548faacc0f nodeName:}" failed. No retries permitted until 2026-10-30 13:40:51.857590434 +0000 UTC m=+2271.004857653 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/44b2314d-1921-47ee-9b8e-65548faacc0f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-74589c-775d48thg" (UID: "44b2314d-1921-47ee-9b8e-65548faacc0f") : secret "agc-cred-587ab343f202" not found
Oct 30 13:40:51.167984 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:40:51.167944    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:40:51.889267 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:40:51.889224    5093 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-74589c/agc-cred-587ab343f202: secret "agc-cred-587ab343f202" not found
Oct 30 13:40:51.889267 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:40:51.889260    5093 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775d48thg: secret "agc-cred-587ab343f202" not found
Oct 30 13:40:51.889509 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:40:51.889331    5093 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44b2314d-1921-47ee-9b8e-65548faacc0f-provider-credentials podName:44b2314d-1921-47ee-9b8e-65548faacc0f nodeName:}" failed. No retries permitted until 2026-10-30 13:40:55.889312137 +0000 UTC m=+2275.036579369 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/44b2314d-1921-47ee-9b8e-65548faacc0f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-74589c-775d48thg" (UID: "44b2314d-1921-47ee-9b8e-65548faacc0f") : secret "agc-cred-587ab343f202" not found
Oct 30 13:40:52.979513 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:52.979472    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-75c46dbdp"]
Oct 30 13:40:55.954611 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:40:55.954574    5093 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-74589c/agc-cred-587ab343f202: secret "agc-cred-587ab343f202" not found
Oct 30 13:40:55.954611 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:40:55.954605    5093 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775d48thg: secret "agc-cred-587ab343f202" not found
Oct 30 13:40:55.955083 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:40:55.954688    5093 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44b2314d-1921-47ee-9b8e-65548faacc0f-provider-credentials podName:44b2314d-1921-47ee-9b8e-65548faacc0f nodeName:}" failed. No retries permitted until 2026-10-30 13:41:03.954666802 +0000 UTC m=+2283.101934038 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/44b2314d-1921-47ee-9b8e-65548faacc0f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-74589c-775d48thg" (UID: "44b2314d-1921-47ee-9b8e-65548faacc0f") : secret "agc-cred-587ab343f202" not found
Oct 30 13:40:57.979850 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:57.979800    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-75c46dbdp" podUID="61f081bf-937f-454f-a1a0-218174acf4c5" containerName="payload-processing" containerID="cri-o://5187ea5e5c6b439a72490884ab2a0f30d48fa7b517bca8fe312ca590b12a5b59" gracePeriod=25
Oct 30 13:40:58.269687 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:58.269654    5093 generic.go:356] "Generic (PLEG): container finished" podID="61f081bf-937f-454f-a1a0-218174acf4c5" containerID="5187ea5e5c6b439a72490884ab2a0f30d48fa7b517bca8fe312ca590b12a5b59" exitCode=0
Oct 30 13:40:58.269854 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:58.269740    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-75c46dbdp" event={"ID":"61f081bf-937f-454f-a1a0-218174acf4c5","Type":"ContainerDied","Data":"5187ea5e5c6b439a72490884ab2a0f30d48fa7b517bca8fe312ca590b12a5b59"}
Oct 30 13:40:58.302583 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:58.302553    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-75c46dbdp"
Oct 30 13:40:58.394599 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:58.394550    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61f081bf-937f-454f-a1a0-218174acf4c5-kube-api-access-wjzl2\" (UniqueName: \"kubernetes.io/projected/61f081bf-937f-454f-a1a0-218174acf4c5-kube-api-access-wjzl2\") pod \"61f081bf-937f-454f-a1a0-218174acf4c5\" (UID: \"61f081bf-937f-454f-a1a0-218174acf4c5\") "
Oct 30 13:40:58.394807 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:58.394679    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61f081bf-937f-454f-a1a0-218174acf4c5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/61f081bf-937f-454f-a1a0-218174acf4c5-provider-credentials\") pod \"61f081bf-937f-454f-a1a0-218174acf4c5\" (UID: \"61f081bf-937f-454f-a1a0-218174acf4c5\") "
Oct 30 13:40:58.394807 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:58.394736    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61f081bf-937f-454f-a1a0-218174acf4c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61f081bf-937f-454f-a1a0-218174acf4c5-plugins-config-volume\") pod \"61f081bf-937f-454f-a1a0-218174acf4c5\" (UID: \"61f081bf-937f-454f-a1a0-218174acf4c5\") "
Oct 30 13:40:58.394947 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:58.394828    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61f081bf-937f-454f-a1a0-218174acf4c5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/61f081bf-937f-454f-a1a0-218174acf4c5-routing-overlay-volume\") pod \"61f081bf-937f-454f-a1a0-218174acf4c5\" (UID: \"61f081bf-937f-454f-a1a0-218174acf4c5\") "
Oct 30 13:40:58.395517 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:58.395451    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61f081bf-937f-454f-a1a0-218174acf4c5-plugins-config-volume" pod "61f081bf-937f-454f-a1a0-218174acf4c5" (UID: "61f081bf-937f-454f-a1a0-218174acf4c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:58.395517 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:58.395471    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61f081bf-937f-454f-a1a0-218174acf4c5-routing-overlay-volume" pod "61f081bf-937f-454f-a1a0-218174acf4c5" (UID: "61f081bf-937f-454f-a1a0-218174acf4c5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:58.397725 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:58.397696    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61f081bf-937f-454f-a1a0-218174acf4c5-kube-api-access-wjzl2" pod "61f081bf-937f-454f-a1a0-218174acf4c5" (UID: "61f081bf-937f-454f-a1a0-218174acf4c5"). InnerVolumeSpecName "kube-api-access-wjzl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:58.397853 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:58.397727    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61f081bf-937f-454f-a1a0-218174acf4c5-provider-credentials" pod "61f081bf-937f-454f-a1a0-218174acf4c5" (UID: "61f081bf-937f-454f-a1a0-218174acf4c5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:58.497440 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:58.497405    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjzl2\" (UniqueName: \"kubernetes.io/projected/61f081bf-937f-454f-a1a0-218174acf4c5-kube-api-access-wjzl2\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:40:58.497440 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:58.497436    5093 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/61f081bf-937f-454f-a1a0-218174acf4c5-provider-credentials\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:40:58.497440 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:58.497448    5093 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61f081bf-937f-454f-a1a0-218174acf4c5-plugins-config-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:40:58.497706 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:58.497456    5093 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/61f081bf-937f-454f-a1a0-218174acf4c5-routing-overlay-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:40:59.280427 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:59.280346    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-75c46dbdp"
Oct 30 13:40:59.280427 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:59.280383    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-75c46dbdp" event={"ID":"61f081bf-937f-454f-a1a0-218174acf4c5","Type":"ContainerDied","Data":"e1b947968ff3a9e8907a64ccca939871421a3be37568598573c2222b6cd6f805"}
Oct 30 13:40:59.280976 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:59.280459    5093 scope.go:122] "RemoveContainer" containerID="5187ea5e5c6b439a72490884ab2a0f30d48fa7b517bca8fe312ca590b12a5b59"
Oct 30 13:40:59.337839 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:59.337801    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-75c46dbdp"]
Oct 30 13:40:59.342146 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:40:59.342101    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-75c46dbdp"]
Oct 30 13:41:00.187745 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:41:00.187714    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775d48thg"
Oct 30 13:41:01.001207 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:41:01.001173    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61f081bf-937f-454f-a1a0-218174acf4c5" path="/var/lib/kubelet/pods/61f081bf-937f-454f-a1a0-218174acf4c5/volumes"
Oct 30 13:41:01.342027 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:41:01.341921    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775d48thg"]
Oct 30 13:41:03.981419 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:41:03.981384    5093 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-74589c/agc-cred-587ab343f202: secret "agc-cred-587ab343f202" not found
Oct 30 13:41:03.981962 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:41:03.981928    5093 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775d48thg: secret "agc-cred-587ab343f202" not found
Oct 30 13:41:03.982098 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:41:03.982021    5093 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44b2314d-1921-47ee-9b8e-65548faacc0f-provider-credentials podName:44b2314d-1921-47ee-9b8e-65548faacc0f nodeName:}" failed. No retries permitted until 2026-10-30 13:41:19.98200037 +0000 UTC m=+2299.129267598 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/44b2314d-1921-47ee-9b8e-65548faacc0f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-74589c-775d48thg" (UID: "44b2314d-1921-47ee-9b8e-65548faacc0f") : secret "agc-cred-587ab343f202" not found
Oct 30 13:41:06.343392 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:41:06.343298    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775d48thg" podUID="44b2314d-1921-47ee-9b8e-65548faacc0f" containerName="payload-processing" containerID="cri-o://617a18df4e378bf68ce1380f23cbcc106e68a5bd8f7e75f9b60b846385aa435a" gracePeriod=25
Oct 30 13:41:06.823417 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:41:06.823377    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775d48thg"
Oct 30 13:41:06.866815 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:41:06.866772    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44b2314d-1921-47ee-9b8e-65548faacc0f-kube-api-access-54gp4\" (UniqueName: \"kubernetes.io/projected/44b2314d-1921-47ee-9b8e-65548faacc0f-kube-api-access-54gp4\") pod \"44b2314d-1921-47ee-9b8e-65548faacc0f\" (UID: \"44b2314d-1921-47ee-9b8e-65548faacc0f\") "
Oct 30 13:41:06.867033 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:41:06.866878    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44b2314d-1921-47ee-9b8e-65548faacc0f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/44b2314d-1921-47ee-9b8e-65548faacc0f-provider-credentials\") pod \"44b2314d-1921-47ee-9b8e-65548faacc0f\" (UID: \"44b2314d-1921-47ee-9b8e-65548faacc0f\") "
Oct 30 13:41:06.867033 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:41:06.866982    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44b2314d-1921-47ee-9b8e-65548faacc0f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44b2314d-1921-47ee-9b8e-65548faacc0f-plugins-config-volume\") pod \"44b2314d-1921-47ee-9b8e-65548faacc0f\" (UID: \"44b2314d-1921-47ee-9b8e-65548faacc0f\") "
Oct 30 13:41:06.867589 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:41:06.867308    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44b2314d-1921-47ee-9b8e-65548faacc0f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/44b2314d-1921-47ee-9b8e-65548faacc0f-routing-overlay-volume\") pod \"44b2314d-1921-47ee-9b8e-65548faacc0f\" (UID: \"44b2314d-1921-47ee-9b8e-65548faacc0f\") "
Oct 30 13:41:06.868863 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:41:06.867948    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44b2314d-1921-47ee-9b8e-65548faacc0f-plugins-config-volume" pod "44b2314d-1921-47ee-9b8e-65548faacc0f" (UID: "44b2314d-1921-47ee-9b8e-65548faacc0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:41:06.869168 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:41:06.869144    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44b2314d-1921-47ee-9b8e-65548faacc0f-routing-overlay-volume" pod "44b2314d-1921-47ee-9b8e-65548faacc0f" (UID: "44b2314d-1921-47ee-9b8e-65548faacc0f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:41:06.872778 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:41:06.872750    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44b2314d-1921-47ee-9b8e-65548faacc0f-kube-api-access-54gp4" pod "44b2314d-1921-47ee-9b8e-65548faacc0f" (UID: "44b2314d-1921-47ee-9b8e-65548faacc0f"). InnerVolumeSpecName "kube-api-access-54gp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:06.878338 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:41:06.878311    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44b2314d-1921-47ee-9b8e-65548faacc0f-provider-credentials" pod "44b2314d-1921-47ee-9b8e-65548faacc0f" (UID: "44b2314d-1921-47ee-9b8e-65548faacc0f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:06.969292 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:41:06.969248    5093 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/44b2314d-1921-47ee-9b8e-65548faacc0f-routing-overlay-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:41:06.969292 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:41:06.969287    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54gp4\" (UniqueName: \"kubernetes.io/projected/44b2314d-1921-47ee-9b8e-65548faacc0f-kube-api-access-54gp4\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:41:06.969569 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:41:06.969300    5093 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/44b2314d-1921-47ee-9b8e-65548faacc0f-provider-credentials\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:41:06.969569 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:41:06.969322    5093 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44b2314d-1921-47ee-9b8e-65548faacc0f-plugins-config-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:41:07.364779 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:41:07.364736    5093 generic.go:356] "Generic (PLEG): container finished" podID="44b2314d-1921-47ee-9b8e-65548faacc0f" containerID="617a18df4e378bf68ce1380f23cbcc106e68a5bd8f7e75f9b60b846385aa435a" exitCode=0
Oct 30 13:41:07.365323 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:41:07.364804    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775d48thg"
Oct 30 13:41:07.365323 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:41:07.365010    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775d48thg" event={"ID":"44b2314d-1921-47ee-9b8e-65548faacc0f","Type":"ContainerDied","Data":"617a18df4e378bf68ce1380f23cbcc106e68a5bd8f7e75f9b60b846385aa435a"}
Oct 30 13:41:07.365323 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:41:07.365060    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775d48thg" event={"ID":"44b2314d-1921-47ee-9b8e-65548faacc0f","Type":"ContainerDied","Data":"22e7f504adc1ce34a6209bba807251eaded5b867f296ea11159bceac1cced98c"}
Oct 30 13:41:07.365323 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:41:07.365090    5093 scope.go:122] "RemoveContainer" containerID="617a18df4e378bf68ce1380f23cbcc106e68a5bd8f7e75f9b60b846385aa435a"
Oct 30 13:41:07.384842 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:41:07.384817    5093 scope.go:122] "RemoveContainer" containerID="617a18df4e378bf68ce1380f23cbcc106e68a5bd8f7e75f9b60b846385aa435a"
Oct 30 13:41:07.385260 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:41:07.385216    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"617a18df4e378bf68ce1380f23cbcc106e68a5bd8f7e75f9b60b846385aa435a\": container with ID starting with 617a18df4e378bf68ce1380f23cbcc106e68a5bd8f7e75f9b60b846385aa435a not found: ID does not exist" containerID="617a18df4e378bf68ce1380f23cbcc106e68a5bd8f7e75f9b60b846385aa435a"
Oct 30 13:41:07.385405 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:41:07.385263    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"617a18df4e378bf68ce1380f23cbcc106e68a5bd8f7e75f9b60b846385aa435a"} err="failed to get container status \"617a18df4e378bf68ce1380f23cbcc106e68a5bd8f7e75f9b60b846385aa435a\": rpc error: code = NotFound desc = could not find container \"617a18df4e378bf68ce1380f23cbcc106e68a5bd8f7e75f9b60b846385aa435a\": container with ID starting with 617a18df4e378bf68ce1380f23cbcc106e68a5bd8f7e75f9b60b846385aa435a not found: ID does not exist"
Oct 30 13:41:07.423945 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:41:07.423911    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775d48thg"]
Oct 30 13:41:07.430032 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:41:07.429986    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-775d48thg"]
Oct 30 13:41:09.002266 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:41:09.002230    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44b2314d-1921-47ee-9b8e-65548faacc0f" path="/var/lib/kubelet/pods/44b2314d-1921-47ee-9b8e-65548faacc0f/volumes"
Oct 30 13:41:21.170169 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:41:21.170095    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:41:51.173462 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:41:51.173416    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:42:21.168736 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:42:21.168686    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:42:43.431549 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:42:43.431456    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v"]
Oct 30 13:42:43.432442 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:42:43.431753    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v" podUID="f9faf395-a37d-4cae-86a5-970702e2b9f1" containerName="istio-proxy" containerID="cri-o://ca30127e78a04dc3a047431a22a397b7f5438bd25dd44be6a007ffcd7a9efd51" gracePeriod=30
Oct 30 13:42:48.730862 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:42:48.730834    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v"
Oct 30 13:42:48.739917 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:42:48.739883    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9faf395-a37d-4cae-86a5-970702e2b9f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9faf395-a37d-4cae-86a5-970702e2b9f1-istiod-ca-cert\") pod \"f9faf395-a37d-4cae-86a5-970702e2b9f1\" (UID: \"f9faf395-a37d-4cae-86a5-970702e2b9f1\") "
Oct 30 13:42:48.740086 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:42:48.739940    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9faf395-a37d-4cae-86a5-970702e2b9f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9faf395-a37d-4cae-86a5-970702e2b9f1-istio-envoy\") pod \"f9faf395-a37d-4cae-86a5-970702e2b9f1\" (UID: \"f9faf395-a37d-4cae-86a5-970702e2b9f1\") "
Oct 30 13:42:48.740086 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:42:48.740030    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9faf395-a37d-4cae-86a5-970702e2b9f1-istio-token\" (UniqueName: \"kubernetes.io/projected/f9faf395-a37d-4cae-86a5-970702e2b9f1-istio-token\") pod \"f9faf395-a37d-4cae-86a5-970702e2b9f1\" (UID: \"f9faf395-a37d-4cae-86a5-970702e2b9f1\") "
Oct 30 13:42:48.740254 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:42:48.740123    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9faf395-a37d-4cae-86a5-970702e2b9f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9faf395-a37d-4cae-86a5-970702e2b9f1-credential-socket\") pod \"f9faf395-a37d-4cae-86a5-970702e2b9f1\" (UID: \"f9faf395-a37d-4cae-86a5-970702e2b9f1\") "
Oct 30 13:42:48.740319 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:42:48.740255    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9faf395-a37d-4cae-86a5-970702e2b9f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9faf395-a37d-4cae-86a5-970702e2b9f1-istio-podinfo\") pod \"f9faf395-a37d-4cae-86a5-970702e2b9f1\" (UID: \"f9faf395-a37d-4cae-86a5-970702e2b9f1\") "
Oct 30 13:42:48.740319 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:42:48.740289    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9faf395-a37d-4cae-86a5-970702e2b9f1-kube-api-access-zwwft\" (UniqueName: \"kubernetes.io/projected/f9faf395-a37d-4cae-86a5-970702e2b9f1-kube-api-access-zwwft\") pod \"f9faf395-a37d-4cae-86a5-970702e2b9f1\" (UID: \"f9faf395-a37d-4cae-86a5-970702e2b9f1\") "
Oct 30 13:42:48.740425 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:42:48.740332    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9faf395-a37d-4cae-86a5-970702e2b9f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9faf395-a37d-4cae-86a5-970702e2b9f1-workload-socket\") pod \"f9faf395-a37d-4cae-86a5-970702e2b9f1\" (UID: \"f9faf395-a37d-4cae-86a5-970702e2b9f1\") "
Oct 30 13:42:48.740425 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:42:48.740394    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9faf395-a37d-4cae-86a5-970702e2b9f1-credential-socket" pod "f9faf395-a37d-4cae-86a5-970702e2b9f1" (UID: "f9faf395-a37d-4cae-86a5-970702e2b9f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:42:48.740425 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:42:48.740407    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9faf395-a37d-4cae-86a5-970702e2b9f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9faf395-a37d-4cae-86a5-970702e2b9f1-istio-data\") pod \"f9faf395-a37d-4cae-86a5-970702e2b9f1\" (UID: \"f9faf395-a37d-4cae-86a5-970702e2b9f1\") "
Oct 30 13:42:48.740598 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:42:48.740444    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9faf395-a37d-4cae-86a5-970702e2b9f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9faf395-a37d-4cae-86a5-970702e2b9f1-workload-certs\") pod \"f9faf395-a37d-4cae-86a5-970702e2b9f1\" (UID: \"f9faf395-a37d-4cae-86a5-970702e2b9f1\") "
Oct 30 13:42:48.740663 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:42:48.740615    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9faf395-a37d-4cae-86a5-970702e2b9f1-workload-socket" pod "f9faf395-a37d-4cae-86a5-970702e2b9f1" (UID: "f9faf395-a37d-4cae-86a5-970702e2b9f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:42:48.740746 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:42:48.740710    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9faf395-a37d-4cae-86a5-970702e2b9f1-istiod-ca-cert" pod "f9faf395-a37d-4cae-86a5-970702e2b9f1" (UID: "f9faf395-a37d-4cae-86a5-970702e2b9f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:42:48.740841 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:42:48.740814    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9faf395-a37d-4cae-86a5-970702e2b9f1-istio-data" pod "f9faf395-a37d-4cae-86a5-970702e2b9f1" (UID: "f9faf395-a37d-4cae-86a5-970702e2b9f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:42:48.740957 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:42:48.740936    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9faf395-a37d-4cae-86a5-970702e2b9f1-workload-certs" pod "f9faf395-a37d-4cae-86a5-970702e2b9f1" (UID: "f9faf395-a37d-4cae-86a5-970702e2b9f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:42:48.741338 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:42:48.741313    5093 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9faf395-a37d-4cae-86a5-970702e2b9f1-credential-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:42:48.741489 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:42:48.741343    5093 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9faf395-a37d-4cae-86a5-970702e2b9f1-workload-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:42:48.741489 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:42:48.741359    5093 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9faf395-a37d-4cae-86a5-970702e2b9f1-istio-data\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:42:48.741489 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:42:48.741373    5093 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9faf395-a37d-4cae-86a5-970702e2b9f1-workload-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:42:48.741489 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:42:48.741386    5093 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9faf395-a37d-4cae-86a5-970702e2b9f1-istiod-ca-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:42:48.742828 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:42:48.742796    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9faf395-a37d-4cae-86a5-970702e2b9f1-istio-envoy" pod "f9faf395-a37d-4cae-86a5-970702e2b9f1" (UID: "f9faf395-a37d-4cae-86a5-970702e2b9f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:42:48.743054 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:42:48.742972    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9faf395-a37d-4cae-86a5-970702e2b9f1-kube-api-access-zwwft" pod "f9faf395-a37d-4cae-86a5-970702e2b9f1" (UID: "f9faf395-a37d-4cae-86a5-970702e2b9f1"). InnerVolumeSpecName "kube-api-access-zwwft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:42:48.743311 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:42:48.743289    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9faf395-a37d-4cae-86a5-970702e2b9f1-istio-podinfo" pod "f9faf395-a37d-4cae-86a5-970702e2b9f1" (UID: "f9faf395-a37d-4cae-86a5-970702e2b9f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:42:48.743466 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:42:48.743447    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9faf395-a37d-4cae-86a5-970702e2b9f1-istio-token" pod "f9faf395-a37d-4cae-86a5-970702e2b9f1" (UID: "f9faf395-a37d-4cae-86a5-970702e2b9f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:42:48.843535 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:42:48.843501    5093 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9faf395-a37d-4cae-86a5-970702e2b9f1-istio-envoy\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:42:48.843535 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:42:48.843531    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9faf395-a37d-4cae-86a5-970702e2b9f1-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:42:48.843535 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:42:48.843539    5093 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9faf395-a37d-4cae-86a5-970702e2b9f1-istio-podinfo\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:42:48.843837 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:42:48.843548    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwwft\" (UniqueName: \"kubernetes.io/projected/f9faf395-a37d-4cae-86a5-970702e2b9f1-kube-api-access-zwwft\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:42:49.468937 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:42:49.468895    5093 generic.go:356] "Generic (PLEG): container finished" podID="f9faf395-a37d-4cae-86a5-970702e2b9f1" containerID="ca30127e78a04dc3a047431a22a397b7f5438bd25dd44be6a007ffcd7a9efd51" exitCode=0
Oct 30 13:42:49.469191 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:42:49.468947    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v"
Oct 30 13:42:49.469191 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:42:49.468970    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v" event={"ID":"f9faf395-a37d-4cae-86a5-970702e2b9f1","Type":"ContainerDied","Data":"ca30127e78a04dc3a047431a22a397b7f5438bd25dd44be6a007ffcd7a9efd51"}
Oct 30 13:42:49.469191 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:42:49.469027    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v" event={"ID":"f9faf395-a37d-4cae-86a5-970702e2b9f1","Type":"ContainerDied","Data":"c24991acbd8d044d3a24773145e82de92b598749ff80445ecb06c3309946d628"}
Oct 30 13:42:49.469191 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:42:49.469054    5093 scope.go:122] "RemoveContainer" containerID="ca30127e78a04dc3a047431a22a397b7f5438bd25dd44be6a007ffcd7a9efd51"
Oct 30 13:42:49.480624 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:42:49.480603    5093 scope.go:122] "RemoveContainer" containerID="ca30127e78a04dc3a047431a22a397b7f5438bd25dd44be6a007ffcd7a9efd51"
Oct 30 13:42:49.480907 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:42:49.480886    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca30127e78a04dc3a047431a22a397b7f5438bd25dd44be6a007ffcd7a9efd51\": container with ID starting with ca30127e78a04dc3a047431a22a397b7f5438bd25dd44be6a007ffcd7a9efd51 not found: ID does not exist" containerID="ca30127e78a04dc3a047431a22a397b7f5438bd25dd44be6a007ffcd7a9efd51"
Oct 30 13:42:49.480956 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:42:49.480917    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca30127e78a04dc3a047431a22a397b7f5438bd25dd44be6a007ffcd7a9efd51"} err="failed to get container status \"ca30127e78a04dc3a047431a22a397b7f5438bd25dd44be6a007ffcd7a9efd51\": rpc error: code = NotFound desc = could not find container \"ca30127e78a04dc3a047431a22a397b7f5438bd25dd44be6a007ffcd7a9efd51\": container with ID starting with ca30127e78a04dc3a047431a22a397b7f5438bd25dd44be6a007ffcd7a9efd51 not found: ID does not exist"
Oct 30 13:42:49.542476 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:42:49.542326    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v"]
Oct 30 13:42:49.556569 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:42:49.556526    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-6bw8v"]
Oct 30 13:42:51.000466 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:42:51.000433    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9faf395-a37d-4cae-86a5-970702e2b9f1" path="/var/lib/kubelet/pods/f9faf395-a37d-4cae-86a5-970702e2b9f1/volumes"
Oct 30 13:42:51.170156 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:42:51.170105    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:43:01.513782 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:43:01.513741    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jdj4_f93ee017-ab79-447c-90e0-5b85b71dba24/console-operator/1.log"
Oct 30 13:43:01.519471 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:43:01.519443    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jdj4_f93ee017-ab79-447c-90e0-5b85b71dba24/console-operator/1.log"
Oct 30 13:43:01.520198 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:43:01.520176    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tnm_3a675ac7-fbe1-449b-accf-bb0456ed0bd3/ovn-acl-logging/0.log"
Oct 30 13:43:01.525877 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:43:01.525851    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tnm_3a675ac7-fbe1-449b-accf-bb0456ed0bd3/ovn-acl-logging/0.log"
Oct 30 13:43:18.662017 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:43:18.661967    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-756d6796bc-ntnj6"]
Oct 30 13:43:18.663044 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:43:18.663016    5093 cpu_manager.go:414] "containerMap: removing container" podUID="61f081bf-937f-454f-a1a0-218174acf4c5" containerName="payload-processing"
Oct 30 13:43:18.663044 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:43:18.663043    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61f081bf-937f-454f-a1a0-218174acf4c5" containerName="payload-processing"
Oct 30 13:43:18.663262 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:43:18.663077    5093 cpu_manager.go:414] "containerMap: removing container" podUID="f9faf395-a37d-4cae-86a5-970702e2b9f1" containerName="istio-proxy"
Oct 30 13:43:18.663262 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:43:18.663085    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9faf395-a37d-4cae-86a5-970702e2b9f1" containerName="istio-proxy"
Oct 30 13:43:18.663262 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:43:18.663156    5093 cpu_manager.go:414] "containerMap: removing container" podUID="44b2314d-1921-47ee-9b8e-65548faacc0f" containerName="payload-processing"
Oct 30 13:43:18.663262 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:43:18.663165    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44b2314d-1921-47ee-9b8e-65548faacc0f" containerName="payload-processing"
Oct 30 13:43:18.663463 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:43:18.663357    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="44b2314d-1921-47ee-9b8e-65548faacc0f" containerName="payload-processing"
Oct 30 13:43:18.663463 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:43:18.663376    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="61f081bf-937f-454f-a1a0-218174acf4c5" containerName="payload-processing"
Oct 30 13:43:18.663463 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:43:18.663391    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9faf395-a37d-4cae-86a5-970702e2b9f1" containerName="istio-proxy"
Oct 30 13:43:18.668841 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:43:18.668811    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-756d6796bc-ntnj6"
Oct 30 13:43:18.685207 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:43:18.685173    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-756d6796bc-ntnj6"]
Oct 30 13:43:18.693536 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:43:18.692761    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-635f8a9a43cd\""
Oct 30 13:43:18.693536 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:43:18.692808    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 13:43:18.693536 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:43:18.692809    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-5vplb\""
Oct 30 13:43:18.693536 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:43:18.692833    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 13:43:18.718485 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:43:18.717892    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 13:43:18.718485 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:43:18.718000    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 13:43:18.830240 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:43:18.830201    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0532d32b-a7b1-4eb3-a0fe-670a54e4db2c-provider-credentials\") pod \"payload-processing-external-model-756d6796bc-ntnj6\" (UID: \"0532d32b-a7b1-4eb3-a0fe-670a54e4db2c\") " pod="models-as-a-service/payload-processing-external-model-756d6796bc-ntnj6"
Oct 30 13:43:18.830708 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:43:18.830677    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0532d32b-a7b1-4eb3-a0fe-670a54e4db2c-plugins-config-volume\") pod \"payload-processing-external-model-756d6796bc-ntnj6\" (UID: \"0532d32b-a7b1-4eb3-a0fe-670a54e4db2c\") " pod="models-as-a-service/payload-processing-external-model-756d6796bc-ntnj6"
Oct 30 13:43:18.830913 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:43:18.830892    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0532d32b-a7b1-4eb3-a0fe-670a54e4db2c-routing-overlay-volume\") pod \"payload-processing-external-model-756d6796bc-ntnj6\" (UID: \"0532d32b-a7b1-4eb3-a0fe-670a54e4db2c\") " pod="models-as-a-service/payload-processing-external-model-756d6796bc-ntnj6"
Oct 30 13:43:18.833095 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:43:18.833075    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nkgj\" (UniqueName: \"kubernetes.io/projected/0532d32b-a7b1-4eb3-a0fe-670a54e4db2c-kube-api-access-4nkgj\") pod \"payload-processing-external-model-756d6796bc-ntnj6\" (UID: \"0532d32b-a7b1-4eb3-a0fe-670a54e4db2c\") " pod="models-as-a-service/payload-processing-external-model-756d6796bc-ntnj6"
Oct 30 13:43:18.935751 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:43:18.935695    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0532d32b-a7b1-4eb3-a0fe-670a54e4db2c-provider-credentials\") pod \"payload-processing-external-model-756d6796bc-ntnj6\" (UID: \"0532d32b-a7b1-4eb3-a0fe-670a54e4db2c\") " pod="models-as-a-service/payload-processing-external-model-756d6796bc-ntnj6"
Oct 30 13:43:18.935965 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:43:18.935915    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0532d32b-a7b1-4eb3-a0fe-670a54e4db2c-plugins-config-volume\") pod \"payload-processing-external-model-756d6796bc-ntnj6\" (UID: \"0532d32b-a7b1-4eb3-a0fe-670a54e4db2c\") " pod="models-as-a-service/payload-processing-external-model-756d6796bc-ntnj6"
Oct 30 13:43:18.935965 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:43:18.935944    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0532d32b-a7b1-4eb3-a0fe-670a54e4db2c-routing-overlay-volume\") pod \"payload-processing-external-model-756d6796bc-ntnj6\" (UID: \"0532d32b-a7b1-4eb3-a0fe-670a54e4db2c\") " pod="models-as-a-service/payload-processing-external-model-756d6796bc-ntnj6"
Oct 30 13:43:18.936145 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:43:18.936114    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nkgj\" (UniqueName: \"kubernetes.io/projected/0532d32b-a7b1-4eb3-a0fe-670a54e4db2c-kube-api-access-4nkgj\") pod \"payload-processing-external-model-756d6796bc-ntnj6\" (UID: \"0532d32b-a7b1-4eb3-a0fe-670a54e4db2c\") " pod="models-as-a-service/payload-processing-external-model-756d6796bc-ntnj6"
Oct 30 13:43:18.936745 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:43:18.936713    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0532d32b-a7b1-4eb3-a0fe-670a54e4db2c-routing-overlay-volume\") pod \"payload-processing-external-model-756d6796bc-ntnj6\" (UID: \"0532d32b-a7b1-4eb3-a0fe-670a54e4db2c\") " pod="models-as-a-service/payload-processing-external-model-756d6796bc-ntnj6"
Oct 30 13:43:18.936876 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:43:18.936826    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0532d32b-a7b1-4eb3-a0fe-670a54e4db2c-plugins-config-volume\") pod \"payload-processing-external-model-756d6796bc-ntnj6\" (UID: \"0532d32b-a7b1-4eb3-a0fe-670a54e4db2c\") " pod="models-as-a-service/payload-processing-external-model-756d6796bc-ntnj6"
Oct 30 13:43:18.939634 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:43:18.939556    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0532d32b-a7b1-4eb3-a0fe-670a54e4db2c-provider-credentials\") pod \"payload-processing-external-model-756d6796bc-ntnj6\" (UID: \"0532d32b-a7b1-4eb3-a0fe-670a54e4db2c\") " pod="models-as-a-service/payload-processing-external-model-756d6796bc-ntnj6"
Oct 30 13:43:18.956914 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:43:18.956876    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nkgj\" (UniqueName: \"kubernetes.io/projected/0532d32b-a7b1-4eb3-a0fe-670a54e4db2c-kube-api-access-4nkgj\") pod \"payload-processing-external-model-756d6796bc-ntnj6\" (UID: \"0532d32b-a7b1-4eb3-a0fe-670a54e4db2c\") " pod="models-as-a-service/payload-processing-external-model-756d6796bc-ntnj6"
Oct 30 13:43:18.986394 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:43:18.986354    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-756d6796bc-ntnj6"
Oct 30 13:43:19.298025 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:43:19.297991    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0532d32b_a7b1_4eb3_a0fe_670a54e4db2c.slice/crio-19c229505899430a8486d30bf02e4410c473bc7c576396dfc769f3693aef55b4 WatchSource:0}: Error finding container 19c229505899430a8486d30bf02e4410c473bc7c576396dfc769f3693aef55b4: Status 404 returned error can't find the container with id 19c229505899430a8486d30bf02e4410c473bc7c576396dfc769f3693aef55b4
Oct 30 13:43:19.299182 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:43:19.299157    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-756d6796bc-ntnj6"]
Oct 30 13:43:19.865530 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:43:19.865492    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-756d6796bc-ntnj6" event={"ID":"0532d32b-a7b1-4eb3-a0fe-670a54e4db2c","Type":"ContainerStarted","Data":"62a02137e656c42f8e4fe234913c5fba427ee5f158f2a29fb091ca593dc48501"}
Oct 30 13:43:19.866032 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:43:19.865547    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-756d6796bc-ntnj6" event={"ID":"0532d32b-a7b1-4eb3-a0fe-670a54e4db2c","Type":"ContainerStarted","Data":"19c229505899430a8486d30bf02e4410c473bc7c576396dfc769f3693aef55b4"}
Oct 30 13:43:19.866032 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:43:19.865636    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-756d6796bc-ntnj6"
Oct 30 13:43:19.911141 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:43:19.911080    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-756d6796bc-ntnj6" podStartSLOduration=1.911055121 podStartE2EDuration="1.911055121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:43:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:43:19.907139513 +0000 UTC m=+2419.054406748" watchObservedRunningTime="2026-10-30 13:43:19.911055121 +0000 UTC m=+2419.058322389"
Oct 30 13:43:21.169705 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:43:21.169665    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:43:30.878810 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:43:30.878774    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-756d6796bc-ntnj6"
Oct 30 13:43:51.172017 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:43:51.171969    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:44:00.859635 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:00.859593    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-7864wbdnh"]
Oct 30 13:44:00.864360 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:00.864332    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-7864wbdnh"
Oct 30 13:44:00.903706 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:00.903668    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-cad2c6\"/\"agc-cred-cf63d4464104\""
Oct 30 13:44:00.903706 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:00.903668    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-cad2c6\"/\"payload-processing-external-model-e2e--8d6d5ae6-dockercfg-7fjrf\""
Oct 30 13:44:00.903706 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:00.903676    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-cad2c6\"/\"agc-cred-6feb6713504c\""
Oct 30 13:44:00.904005 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:00.903759    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cad2c6\"/\"openshift-service-ca.crt\""
Oct 30 13:44:00.904005 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:00.903796    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cad2c6\"/\"payload-processing-external-model-plugins-e2e-extmod-a-cad2c6\""
Oct 30 13:44:00.904005 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:00.903964    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cad2c6\"/\"routing-overlay\""
Oct 30 13:44:00.904177 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:00.904030    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cad2c6\"/\"kube-root-ca.crt\""
Oct 30 13:44:00.909078 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:00.909051    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-7864wbdnh"]
Oct 30 13:44:00.996716 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:00.996448    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6641c45f-b1ee-4de8-98d8-dd17093cda57-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-7864wbdnh\" (UID: \"6641c45f-b1ee-4de8-98d8-dd17093cda57\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-7864wbdnh"
Oct 30 13:44:00.996716 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:00.996546    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mptrk\" (UniqueName: \"kubernetes.io/projected/6641c45f-b1ee-4de8-98d8-dd17093cda57-kube-api-access-mptrk\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-7864wbdnh\" (UID: \"6641c45f-b1ee-4de8-98d8-dd17093cda57\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-7864wbdnh"
Oct 30 13:44:00.996716 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:00.996606    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6641c45f-b1ee-4de8-98d8-dd17093cda57-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-7864wbdnh\" (UID: \"6641c45f-b1ee-4de8-98d8-dd17093cda57\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-7864wbdnh"
Oct 30 13:44:00.997041 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:00.996855    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6641c45f-b1ee-4de8-98d8-dd17093cda57-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-7864wbdnh\" (UID: \"6641c45f-b1ee-4de8-98d8-dd17093cda57\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-7864wbdnh"
Oct 30 13:44:01.097875 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:01.097827    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6641c45f-b1ee-4de8-98d8-dd17093cda57-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-7864wbdnh\" (UID: \"6641c45f-b1ee-4de8-98d8-dd17093cda57\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-7864wbdnh"
Oct 30 13:44:01.098109 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:01.097972    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6641c45f-b1ee-4de8-98d8-dd17093cda57-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-7864wbdnh\" (UID: \"6641c45f-b1ee-4de8-98d8-dd17093cda57\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-7864wbdnh"
Oct 30 13:44:01.098109 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:01.098053    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mptrk\" (UniqueName: \"kubernetes.io/projected/6641c45f-b1ee-4de8-98d8-dd17093cda57-kube-api-access-mptrk\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-7864wbdnh\" (UID: \"6641c45f-b1ee-4de8-98d8-dd17093cda57\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-7864wbdnh"
Oct 30 13:44:01.098346 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:01.098146    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6641c45f-b1ee-4de8-98d8-dd17093cda57-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-7864wbdnh\" (UID: \"6641c45f-b1ee-4de8-98d8-dd17093cda57\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-7864wbdnh"
Oct 30 13:44:01.113302 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:01.113213    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cad2c6\"/\"payload-processing-external-model-plugins-e2e-extmod-a-cad2c6\""
Oct 30 13:44:01.113430 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:01.113313    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cad2c6\"/\"routing-overlay\""
Oct 30 13:44:01.113430 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:01.113348    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-cad2c6\"/\"agc-cred-6feb6713504c\""
Oct 30 13:44:01.118727 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:01.118703    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6641c45f-b1ee-4de8-98d8-dd17093cda57-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-7864wbdnh\" (UID: \"6641c45f-b1ee-4de8-98d8-dd17093cda57\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-7864wbdnh"
Oct 30 13:44:01.119391 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:01.119369    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6641c45f-b1ee-4de8-98d8-dd17093cda57-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-7864wbdnh\" (UID: \"6641c45f-b1ee-4de8-98d8-dd17093cda57\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-7864wbdnh"
Oct 30 13:44:01.132091 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:01.132065    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cad2c6\"/\"kube-root-ca.crt\""
Oct 30 13:44:01.132555 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:01.132535    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-cad2c6\"/\"agc-cred-cf63d4464104\""
Oct 30 13:44:01.141318 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:01.141291    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6641c45f-b1ee-4de8-98d8-dd17093cda57-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-7864wbdnh\" (UID: \"6641c45f-b1ee-4de8-98d8-dd17093cda57\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-7864wbdnh"
Oct 30 13:44:01.153246 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:01.153218    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cad2c6\"/\"openshift-service-ca.crt\""
Oct 30 13:44:01.160937 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:01.160894    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mptrk\" (UniqueName: \"kubernetes.io/projected/6641c45f-b1ee-4de8-98d8-dd17093cda57-kube-api-access-mptrk\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-7864wbdnh\" (UID: \"6641c45f-b1ee-4de8-98d8-dd17093cda57\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-7864wbdnh"
Oct 30 13:44:01.198576 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:01.198534    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-cad2c6\"/\"payload-processing-external-model-e2e--8d6d5ae6-dockercfg-7fjrf\""
Oct 30 13:44:01.204652 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:01.204621    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-7864wbdnh"
Oct 30 13:44:01.527664 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:01.527634    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-7864wbdnh"]
Oct 30 13:44:01.528597 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:44:01.528572    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6641c45f_b1ee_4de8_98d8_dd17093cda57.slice/crio-232296df6afefd72fe9dc2305c049a0416f3be491b6c7606816cff2d5e5916ce WatchSource:0}: Error finding container 232296df6afefd72fe9dc2305c049a0416f3be491b6c7606816cff2d5e5916ce: Status 404 returned error can't find the container with id 232296df6afefd72fe9dc2305c049a0416f3be491b6c7606816cff2d5e5916ce
Oct 30 13:44:02.297542 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:02.297502    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-7864wbdnh" event={"ID":"6641c45f-b1ee-4de8-98d8-dd17093cda57","Type":"ContainerStarted","Data":"824e4add8d144638fc2f1aa7d5766423ca3e77639423d8251c6c3b93aad37fa3"}
Oct 30 13:44:02.298156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:02.297568    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-7864wbdnh" event={"ID":"6641c45f-b1ee-4de8-98d8-dd17093cda57","Type":"ContainerStarted","Data":"232296df6afefd72fe9dc2305c049a0416f3be491b6c7606816cff2d5e5916ce"}
Oct 30 13:44:02.298156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:02.297595    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-7864wbdnh"
Oct 30 13:44:02.348870 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:02.348808    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-7864wbdnh" podStartSLOduration=2.348786388 podStartE2EDuration="2.348786388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:44:02.346074569 +0000 UTC m=+2461.493341810" watchObservedRunningTime="2026-10-30 13:44:02.348786388 +0000 UTC m=+2461.496053642"
Oct 30 13:44:12.653333 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:12.653081    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-756d6796bc-ntnj6"]
Oct 30 13:44:13.319865 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:13.319820    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-7864wbdnh"
Oct 30 13:44:15.701014 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:15.700969    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h"]
Oct 30 13:44:15.706163 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:15.706099    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h"
Oct 30 13:44:15.741107 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:15.741068    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f84ad73e-12f3-4784-9928-bd6a24257682-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-7xk7h\" (UID: \"f84ad73e-12f3-4784-9928-bd6a24257682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h"
Oct 30 13:44:15.741344 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:15.741193    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f84ad73e-12f3-4784-9928-bd6a24257682-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-7xk7h\" (UID: \"f84ad73e-12f3-4784-9928-bd6a24257682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h"
Oct 30 13:44:15.741344 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:15.741238    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8v8l\" (UniqueName: \"kubernetes.io/projected/f84ad73e-12f3-4784-9928-bd6a24257682-kube-api-access-p8v8l\") pod \"maas-default-gateway-openshift-default-67bf986f99-7xk7h\" (UID: \"f84ad73e-12f3-4784-9928-bd6a24257682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h"
Oct 30 13:44:15.741470 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:15.741344    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84ad73e-12f3-4784-9928-bd6a24257682-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-7xk7h\" (UID: \"f84ad73e-12f3-4784-9928-bd6a24257682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h"
Oct 30 13:44:15.741470 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:15.741392    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f84ad73e-12f3-4784-9928-bd6a24257682-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-7xk7h\" (UID: \"f84ad73e-12f3-4784-9928-bd6a24257682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h"
Oct 30 13:44:15.741470 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:15.741459    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84ad73e-12f3-4784-9928-bd6a24257682-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-7xk7h\" (UID: \"f84ad73e-12f3-4784-9928-bd6a24257682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h"
Oct 30 13:44:15.741627 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:15.741491    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f84ad73e-12f3-4784-9928-bd6a24257682-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-7xk7h\" (UID: \"f84ad73e-12f3-4784-9928-bd6a24257682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h"
Oct 30 13:44:15.741627 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:15.741578    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h"]
Oct 30 13:44:15.741732 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:15.741584    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f84ad73e-12f3-4784-9928-bd6a24257682-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-7xk7h\" (UID: \"f84ad73e-12f3-4784-9928-bd6a24257682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h"
Oct 30 13:44:15.741786 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:15.741743    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f84ad73e-12f3-4784-9928-bd6a24257682-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-7xk7h\" (UID: \"f84ad73e-12f3-4784-9928-bd6a24257682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h"
Oct 30 13:44:15.843507 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:15.843470    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f84ad73e-12f3-4784-9928-bd6a24257682-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-7xk7h\" (UID: \"f84ad73e-12f3-4784-9928-bd6a24257682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h"
Oct 30 13:44:15.843713 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:15.843515    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8v8l\" (UniqueName: \"kubernetes.io/projected/f84ad73e-12f3-4784-9928-bd6a24257682-kube-api-access-p8v8l\") pod \"maas-default-gateway-openshift-default-67bf986f99-7xk7h\" (UID: \"f84ad73e-12f3-4784-9928-bd6a24257682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h"
Oct 30 13:44:15.843713 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:15.843554    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84ad73e-12f3-4784-9928-bd6a24257682-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-7xk7h\" (UID: \"f84ad73e-12f3-4784-9928-bd6a24257682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h"
Oct 30 13:44:15.843713 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:15.843584    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f84ad73e-12f3-4784-9928-bd6a24257682-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-7xk7h\" (UID: \"f84ad73e-12f3-4784-9928-bd6a24257682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h"
Oct 30 13:44:15.843904 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:15.843725    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84ad73e-12f3-4784-9928-bd6a24257682-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-7xk7h\" (UID: \"f84ad73e-12f3-4784-9928-bd6a24257682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h"
Oct 30 13:44:15.843904 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:15.843788    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f84ad73e-12f3-4784-9928-bd6a24257682-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-7xk7h\" (UID: \"f84ad73e-12f3-4784-9928-bd6a24257682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h"
Oct 30 13:44:15.844061 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:15.843899    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f84ad73e-12f3-4784-9928-bd6a24257682-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-7xk7h\" (UID: \"f84ad73e-12f3-4784-9928-bd6a24257682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h"
Oct 30 13:44:15.844061 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:15.843974    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84ad73e-12f3-4784-9928-bd6a24257682-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-7xk7h\" (UID: \"f84ad73e-12f3-4784-9928-bd6a24257682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h"
Oct 30 13:44:15.844061 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:15.844008    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f84ad73e-12f3-4784-9928-bd6a24257682-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-7xk7h\" (UID: \"f84ad73e-12f3-4784-9928-bd6a24257682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h"
Oct 30 13:44:15.844061 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:15.844045    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f84ad73e-12f3-4784-9928-bd6a24257682-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-7xk7h\" (UID: \"f84ad73e-12f3-4784-9928-bd6a24257682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h"
Oct 30 13:44:15.844327 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:15.844006    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f84ad73e-12f3-4784-9928-bd6a24257682-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-7xk7h\" (UID: \"f84ad73e-12f3-4784-9928-bd6a24257682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h"
Oct 30 13:44:15.844327 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:15.844177    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f84ad73e-12f3-4784-9928-bd6a24257682-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-7xk7h\" (UID: \"f84ad73e-12f3-4784-9928-bd6a24257682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h"
Oct 30 13:44:15.844471 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:15.844445    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84ad73e-12f3-4784-9928-bd6a24257682-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-7xk7h\" (UID: \"f84ad73e-12f3-4784-9928-bd6a24257682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h"
Oct 30 13:44:15.844693 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:15.844664    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f84ad73e-12f3-4784-9928-bd6a24257682-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-7xk7h\" (UID: \"f84ad73e-12f3-4784-9928-bd6a24257682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h"
Oct 30 13:44:15.846653 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:15.846625    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f84ad73e-12f3-4784-9928-bd6a24257682-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-7xk7h\" (UID: \"f84ad73e-12f3-4784-9928-bd6a24257682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h"
Oct 30 13:44:15.846883 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:15.846861    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f84ad73e-12f3-4784-9928-bd6a24257682-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-7xk7h\" (UID: \"f84ad73e-12f3-4784-9928-bd6a24257682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h"
Oct 30 13:44:15.867093 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:15.867059    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8v8l\" (UniqueName: \"kubernetes.io/projected/f84ad73e-12f3-4784-9928-bd6a24257682-kube-api-access-p8v8l\") pod \"maas-default-gateway-openshift-default-67bf986f99-7xk7h\" (UID: \"f84ad73e-12f3-4784-9928-bd6a24257682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h"
Oct 30 13:44:15.867547 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:15.867525    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f84ad73e-12f3-4784-9928-bd6a24257682-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-7xk7h\" (UID: \"f84ad73e-12f3-4784-9928-bd6a24257682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h"
Oct 30 13:44:16.039423 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:16.039338    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h"
Oct 30 13:44:16.381732 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:16.381700    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h"]
Oct 30 13:44:16.383881 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:44:16.383853    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf84ad73e_12f3_4784_9928_bd6a24257682.slice/crio-e53f233a7e7db7f392ed32be8991401f59487563ceab9de1e7792db75c68c2e8 WatchSource:0}: Error finding container e53f233a7e7db7f392ed32be8991401f59487563ceab9de1e7792db75c68c2e8: Status 404 returned error can't find the container with id e53f233a7e7db7f392ed32be8991401f59487563ceab9de1e7792db75c68c2e8
Oct 30 13:44:16.387682 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:16.387648    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:44:16.387794 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:16.387719    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:44:16.387794 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:16.387750    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:44:16.437726 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:16.437617    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h" event={"ID":"f84ad73e-12f3-4784-9928-bd6a24257682","Type":"ContainerStarted","Data":"e53f233a7e7db7f392ed32be8991401f59487563ceab9de1e7792db75c68c2e8"}
Oct 30 13:44:17.506151 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:17.505841    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h" event={"ID":"f84ad73e-12f3-4784-9928-bd6a24257682","Type":"ContainerStarted","Data":"c44220484b3d3c73e15674a80cf7ea18d00b5e59910f94bbcf740e6e0d083c45"}
Oct 30 13:44:17.592791 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:17.592722    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h" podStartSLOduration=2.59270756 podStartE2EDuration="2.59270756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:44:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:44:17.576326538 +0000 UTC m=+2476.723593771" watchObservedRunningTime="2026-10-30 13:44:17.59270756 +0000 UTC m=+2476.739974800"
Oct 30 13:44:17.655061 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:17.654359    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-756d6796bc-ntnj6" podUID="0532d32b-a7b1-4eb3-a0fe-670a54e4db2c" containerName="payload-processing" containerID="cri-o://62a02137e656c42f8e4fe234913c5fba427ee5f158f2a29fb091ca593dc48501" gracePeriod=25
Oct 30 13:44:18.056494 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:18.056280    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h"
Oct 30 13:44:18.059363 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:18.058822    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-7xk7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 30 13:44:18.059363 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:18.058881    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h" podUID="f84ad73e-12f3-4784-9928-bd6a24257682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 30 13:44:18.064559 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:18.064540    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-756d6796bc-ntnj6"
Oct 30 13:44:18.199838 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:18.199806    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0532d32b-a7b1-4eb3-a0fe-670a54e4db2c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0532d32b-a7b1-4eb3-a0fe-670a54e4db2c-routing-overlay-volume\") pod \"0532d32b-a7b1-4eb3-a0fe-670a54e4db2c\" (UID: \"0532d32b-a7b1-4eb3-a0fe-670a54e4db2c\") "
Oct 30 13:44:18.200020 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:18.199930    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0532d32b-a7b1-4eb3-a0fe-670a54e4db2c-kube-api-access-4nkgj\" (UniqueName: \"kubernetes.io/projected/0532d32b-a7b1-4eb3-a0fe-670a54e4db2c-kube-api-access-4nkgj\") pod \"0532d32b-a7b1-4eb3-a0fe-670a54e4db2c\" (UID: \"0532d32b-a7b1-4eb3-a0fe-670a54e4db2c\") "
Oct 30 13:44:18.200020 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:18.199959    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0532d32b-a7b1-4eb3-a0fe-670a54e4db2c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0532d32b-a7b1-4eb3-a0fe-670a54e4db2c-plugins-config-volume\") pod \"0532d32b-a7b1-4eb3-a0fe-670a54e4db2c\" (UID: \"0532d32b-a7b1-4eb3-a0fe-670a54e4db2c\") "
Oct 30 13:44:18.200020 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:18.199988    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0532d32b-a7b1-4eb3-a0fe-670a54e4db2c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0532d32b-a7b1-4eb3-a0fe-670a54e4db2c-provider-credentials\") pod \"0532d32b-a7b1-4eb3-a0fe-670a54e4db2c\" (UID: \"0532d32b-a7b1-4eb3-a0fe-670a54e4db2c\") "
Oct 30 13:44:18.200334 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:18.200307    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0532d32b-a7b1-4eb3-a0fe-670a54e4db2c-routing-overlay-volume" pod "0532d32b-a7b1-4eb3-a0fe-670a54e4db2c" (UID: "0532d32b-a7b1-4eb3-a0fe-670a54e4db2c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:44:18.200507 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:18.200479    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0532d32b-a7b1-4eb3-a0fe-670a54e4db2c-plugins-config-volume" pod "0532d32b-a7b1-4eb3-a0fe-670a54e4db2c" (UID: "0532d32b-a7b1-4eb3-a0fe-670a54e4db2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:44:18.200763 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:18.200734    5093 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0532d32b-a7b1-4eb3-a0fe-670a54e4db2c-routing-overlay-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:44:18.200888 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:18.200766    5093 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0532d32b-a7b1-4eb3-a0fe-670a54e4db2c-plugins-config-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:44:18.202343 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:18.202321    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0532d32b-a7b1-4eb3-a0fe-670a54e4db2c-kube-api-access-4nkgj" pod "0532d32b-a7b1-4eb3-a0fe-670a54e4db2c" (UID: "0532d32b-a7b1-4eb3-a0fe-670a54e4db2c"). InnerVolumeSpecName "kube-api-access-4nkgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:18.202444 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:18.202400    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0532d32b-a7b1-4eb3-a0fe-670a54e4db2c-provider-credentials" pod "0532d32b-a7b1-4eb3-a0fe-670a54e4db2c" (UID: "0532d32b-a7b1-4eb3-a0fe-670a54e4db2c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:18.302485 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:18.302446    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nkgj\" (UniqueName: \"kubernetes.io/projected/0532d32b-a7b1-4eb3-a0fe-670a54e4db2c-kube-api-access-4nkgj\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:44:18.302485 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:18.302485    5093 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0532d32b-a7b1-4eb3-a0fe-670a54e4db2c-provider-credentials\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:44:18.516125 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:18.516079    5093 generic.go:356] "Generic (PLEG): container finished" podID="0532d32b-a7b1-4eb3-a0fe-670a54e4db2c" containerID="62a02137e656c42f8e4fe234913c5fba427ee5f158f2a29fb091ca593dc48501" exitCode=0
Oct 30 13:44:18.516125 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:18.516122    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-756d6796bc-ntnj6" event={"ID":"0532d32b-a7b1-4eb3-a0fe-670a54e4db2c","Type":"ContainerDied","Data":"62a02137e656c42f8e4fe234913c5fba427ee5f158f2a29fb091ca593dc48501"}
Oct 30 13:44:18.516715 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:18.516214    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-756d6796bc-ntnj6" event={"ID":"0532d32b-a7b1-4eb3-a0fe-670a54e4db2c","Type":"ContainerDied","Data":"19c229505899430a8486d30bf02e4410c473bc7c576396dfc769f3693aef55b4"}
Oct 30 13:44:18.516715 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:18.516241    5093 scope.go:122] "RemoveContainer" containerID="62a02137e656c42f8e4fe234913c5fba427ee5f158f2a29fb091ca593dc48501"
Oct 30 13:44:18.516715 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:18.516163    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-756d6796bc-ntnj6"
Oct 30 13:44:18.557298 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:18.557260    5093 scope.go:122] "RemoveContainer" containerID="62a02137e656c42f8e4fe234913c5fba427ee5f158f2a29fb091ca593dc48501"
Oct 30 13:44:18.557747 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:44:18.557716    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62a02137e656c42f8e4fe234913c5fba427ee5f158f2a29fb091ca593dc48501\": container with ID starting with 62a02137e656c42f8e4fe234913c5fba427ee5f158f2a29fb091ca593dc48501 not found: ID does not exist" containerID="62a02137e656c42f8e4fe234913c5fba427ee5f158f2a29fb091ca593dc48501"
Oct 30 13:44:18.557883 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:18.557759    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62a02137e656c42f8e4fe234913c5fba427ee5f158f2a29fb091ca593dc48501"} err="failed to get container status \"62a02137e656c42f8e4fe234913c5fba427ee5f158f2a29fb091ca593dc48501\": rpc error: code = NotFound desc = could not find container \"62a02137e656c42f8e4fe234913c5fba427ee5f158f2a29fb091ca593dc48501\": container with ID starting with 62a02137e656c42f8e4fe234913c5fba427ee5f158f2a29fb091ca593dc48501 not found: ID does not exist"
Oct 30 13:44:18.590791 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:18.590753    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-756d6796bc-ntnj6"]
Oct 30 13:44:18.601456 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:18.601422    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-756d6796bc-ntnj6"]
Oct 30 13:44:19.002213 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:19.002183    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0532d32b-a7b1-4eb3-a0fe-670a54e4db2c" path="/var/lib/kubelet/pods/0532d32b-a7b1-4eb3-a0fe-670a54e4db2c/volumes"
Oct 30 13:44:19.042151 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:19.040534    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-7xk7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 30 13:44:19.042151 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:19.040611    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h" podUID="f84ad73e-12f3-4784-9928-bd6a24257682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 30 13:44:20.044167 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:20.043537    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-7xk7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 30 13:44:20.044167 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:20.043602    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h" podUID="f84ad73e-12f3-4784-9928-bd6a24257682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 30 13:44:21.040174 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:21.039751    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-7xk7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 30 13:44:21.040174 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:21.039818    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h" podUID="f84ad73e-12f3-4784-9928-bd6a24257682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 30 13:44:21.229934 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:44:21.229857    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:44:22.049636 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:22.049587    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-7xk7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 30 13:44:22.049938 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:22.049904    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h" podUID="f84ad73e-12f3-4784-9928-bd6a24257682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 30 13:44:23.040508 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:23.040463    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-7xk7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 30 13:44:23.041023 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:23.040529    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h" podUID="f84ad73e-12f3-4784-9928-bd6a24257682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 30 13:44:24.040033 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:24.039989    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-7xk7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 30 13:44:24.040252 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:24.040063    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h" podUID="f84ad73e-12f3-4784-9928-bd6a24257682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 30 13:44:26.040221 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:26.040179    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-7xk7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:44:26.040870 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:26.040825    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h" podUID="f84ad73e-12f3-4784-9928-bd6a24257682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:44:27.082299 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:27.082254    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-7xk7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:44:27.082822 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:27.082338    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h" podUID="f84ad73e-12f3-4784-9928-bd6a24257682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:44:27.849974 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:27.849936    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h"
Oct 30 13:44:27.850443 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:27.850420    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h"
Oct 30 13:44:27.851097 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:27.851079    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h"
Oct 30 13:44:34.738149 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:34.735789    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9sgpb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:44:34.738149 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:44:34.735872    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb" podUID="e97d49cb-0e82-4c03-8aeb-044f1c076234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:44:51.169683 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:44:51.169636    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:45:00.407277 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:00.407237    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889465-sfznh"]
Oct 30 13:45:00.408421 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:00.408213    5093 cpu_manager.go:414] "containerMap: removing container" podUID="0532d32b-a7b1-4eb3-a0fe-670a54e4db2c" containerName="payload-processing"
Oct 30 13:45:00.408421 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:00.408236    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0532d32b-a7b1-4eb3-a0fe-670a54e4db2c" containerName="payload-processing"
Oct 30 13:45:00.408421 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:00.408376    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="0532d32b-a7b1-4eb3-a0fe-670a54e4db2c" containerName="payload-processing"
Oct 30 13:45:00.412461 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:00.412433    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889465-sfznh"
Oct 30 13:45:00.417389 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:00.417353    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889465-r98tc"]
Oct 30 13:45:00.423336 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:00.423304    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889465-r98tc"
Oct 30 13:45:00.449166 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:00.442319    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889465-qtb2r"]
Oct 30 13:45:00.449166 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:00.448829    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889465-qtb2r"
Oct 30 13:45:00.471149 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:00.461249    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889465-sfznh"]
Oct 30 13:45:00.471149 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:00.461411    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889465-r98tc"]
Oct 30 13:45:00.474994 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:00.474966    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889465-6j5w8"]
Oct 30 13:45:00.481121 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:00.481089    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889465-6j5w8"
Oct 30 13:45:00.485967 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:00.485923    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:45:00.490266 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:00.490236    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889465-qtb2r"]
Oct 30 13:45:00.506935 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:00.506892    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889465-6j5w8"]
Oct 30 13:45:00.525577 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:00.525539    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38d0b166-12f0-4fab-9dd8-934f86ef2b91-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5852ff-29889465-r98tc\" (UID: \"38d0b166-12f0-4fab-9dd8-934f86ef2b91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889465-r98tc"
Oct 30 13:45:00.525776 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:00.525651    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4l9j\" (UniqueName: \"kubernetes.io/projected/38d0b166-12f0-4fab-9dd8-934f86ef2b91-kube-api-access-f4l9j\") pod \"maas-api-key-cleanup-e2e-worker-w3-5852ff-29889465-r98tc\" (UID: \"38d0b166-12f0-4fab-9dd8-934f86ef2b91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889465-r98tc"
Oct 30 13:45:00.525776 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:00.525694    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76fck\" (UniqueName: \"kubernetes.io/projected/6425c367-ad27-4b67-bace-4b94ff4169fc-kube-api-access-76fck\") pod \"maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889465-qtb2r\" (UID: \"6425c367-ad27-4b67-bace-4b94ff4169fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889465-qtb2r"
Oct 30 13:45:00.525776 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:00.525715    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6425c367-ad27-4b67-bace-4b94ff4169fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889465-qtb2r\" (UID: \"6425c367-ad27-4b67-bace-4b94ff4169fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889465-qtb2r"
Oct 30 13:45:00.525776 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:00.525774    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px9vq\" (UniqueName: \"kubernetes.io/projected/66aff1fb-d9e0-463e-81ae-71da54df6955-kube-api-access-px9vq\") pod \"maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889465-sfznh\" (UID: \"66aff1fb-d9e0-463e-81ae-71da54df6955\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889465-sfznh"
Oct 30 13:45:00.526025 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:00.525820    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66aff1fb-d9e0-463e-81ae-71da54df6955-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889465-sfznh\" (UID: \"66aff1fb-d9e0-463e-81ae-71da54df6955\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889465-sfznh"
Oct 30 13:45:00.627042 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:00.626997    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px9vq\" (UniqueName: \"kubernetes.io/projected/66aff1fb-d9e0-463e-81ae-71da54df6955-kube-api-access-px9vq\") pod \"maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889465-sfznh\" (UID: \"66aff1fb-d9e0-463e-81ae-71da54df6955\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889465-sfznh"
Oct 30 13:45:00.627281 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:00.627103    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66aff1fb-d9e0-463e-81ae-71da54df6955-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889465-sfznh\" (UID: \"66aff1fb-d9e0-463e-81ae-71da54df6955\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889465-sfznh"
Oct 30 13:45:00.627281 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:00.627209    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd6n7\" (UniqueName: \"kubernetes.io/projected/0b6f42b5-10bf-4920-9acb-7e37f70f99e7-kube-api-access-vd6n7\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889465-6j5w8\" (UID: \"0b6f42b5-10bf-4920-9acb-7e37f70f99e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889465-6j5w8"
Oct 30 13:45:00.627424 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:00.627378    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38d0b166-12f0-4fab-9dd8-934f86ef2b91-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5852ff-29889465-r98tc\" (UID: \"38d0b166-12f0-4fab-9dd8-934f86ef2b91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889465-r98tc"
Oct 30 13:45:00.627569 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:00.627551    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4l9j\" (UniqueName: \"kubernetes.io/projected/38d0b166-12f0-4fab-9dd8-934f86ef2b91-kube-api-access-f4l9j\") pod \"maas-api-key-cleanup-e2e-worker-w3-5852ff-29889465-r98tc\" (UID: \"38d0b166-12f0-4fab-9dd8-934f86ef2b91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889465-r98tc"
Oct 30 13:45:00.627632 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:00.627609    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76fck\" (UniqueName: \"kubernetes.io/projected/6425c367-ad27-4b67-bace-4b94ff4169fc-kube-api-access-76fck\") pod \"maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889465-qtb2r\" (UID: \"6425c367-ad27-4b67-bace-4b94ff4169fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889465-qtb2r"
Oct 30 13:45:00.627693 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:00.627644    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b6f42b5-10bf-4920-9acb-7e37f70f99e7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889465-6j5w8\" (UID: \"0b6f42b5-10bf-4920-9acb-7e37f70f99e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889465-6j5w8"
Oct 30 13:45:00.627693 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:00.627677    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6425c367-ad27-4b67-bace-4b94ff4169fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889465-qtb2r\" (UID: \"6425c367-ad27-4b67-bace-4b94ff4169fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889465-qtb2r"
Oct 30 13:45:00.628024 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:00.628001    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66aff1fb-d9e0-463e-81ae-71da54df6955-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889465-sfznh\" (UID: \"66aff1fb-d9e0-463e-81ae-71da54df6955\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889465-sfznh"
Oct 30 13:45:00.628295 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:00.628273    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38d0b166-12f0-4fab-9dd8-934f86ef2b91-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5852ff-29889465-r98tc\" (UID: \"38d0b166-12f0-4fab-9dd8-934f86ef2b91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889465-r98tc"
Oct 30 13:45:00.628430 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:00.628413    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6425c367-ad27-4b67-bace-4b94ff4169fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889465-qtb2r\" (UID: \"6425c367-ad27-4b67-bace-4b94ff4169fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889465-qtb2r"
Oct 30 13:45:00.649251 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:00.649215    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76fck\" (UniqueName: \"kubernetes.io/projected/6425c367-ad27-4b67-bace-4b94ff4169fc-kube-api-access-76fck\") pod \"maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889465-qtb2r\" (UID: \"6425c367-ad27-4b67-bace-4b94ff4169fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889465-qtb2r"
Oct 30 13:45:00.649558 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:00.649530    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px9vq\" (UniqueName: \"kubernetes.io/projected/66aff1fb-d9e0-463e-81ae-71da54df6955-kube-api-access-px9vq\") pod \"maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889465-sfznh\" (UID: \"66aff1fb-d9e0-463e-81ae-71da54df6955\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889465-sfznh"
Oct 30 13:45:00.649616 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:00.649537    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4l9j\" (UniqueName: \"kubernetes.io/projected/38d0b166-12f0-4fab-9dd8-934f86ef2b91-kube-api-access-f4l9j\") pod \"maas-api-key-cleanup-e2e-worker-w3-5852ff-29889465-r98tc\" (UID: \"38d0b166-12f0-4fab-9dd8-934f86ef2b91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889465-r98tc"
Oct 30 13:45:00.729516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:00.729460    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b6f42b5-10bf-4920-9acb-7e37f70f99e7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889465-6j5w8\" (UID: \"0b6f42b5-10bf-4920-9acb-7e37f70f99e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889465-6j5w8"
Oct 30 13:45:00.729751 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:00.729683    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd6n7\" (UniqueName: \"kubernetes.io/projected/0b6f42b5-10bf-4920-9acb-7e37f70f99e7-kube-api-access-vd6n7\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889465-6j5w8\" (UID: \"0b6f42b5-10bf-4920-9acb-7e37f70f99e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889465-6j5w8"
Oct 30 13:45:00.730235 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:00.730205    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b6f42b5-10bf-4920-9acb-7e37f70f99e7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889465-6j5w8\" (UID: \"0b6f42b5-10bf-4920-9acb-7e37f70f99e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889465-6j5w8"
Oct 30 13:45:00.755636 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:00.755591    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd6n7\" (UniqueName: \"kubernetes.io/projected/0b6f42b5-10bf-4920-9acb-7e37f70f99e7-kube-api-access-vd6n7\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889465-6j5w8\" (UID: \"0b6f42b5-10bf-4920-9acb-7e37f70f99e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889465-6j5w8"
Oct 30 13:45:00.763170 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:00.763121    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889465-r98tc"
Oct 30 13:45:00.778652 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:00.778616    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889465-sfznh"
Oct 30 13:45:00.783031 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:00.783000    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889465-qtb2r"
Oct 30 13:45:00.809207 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:00.809179    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889465-6j5w8"
Oct 30 13:45:00.887633 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:00.887605    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c"]
Oct 30 13:45:00.913005 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:00.911410    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c"
Oct 30 13:45:00.918669 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:00.918641    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c"]
Oct 30 13:45:01.038634 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:01.037357    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-hbc8c\" (UID: \"d6d8c234-2d8c-488b-9ea9-3969e9c430bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c"
Oct 30 13:45:01.038634 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:01.037449    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-hbc8c\" (UID: \"d6d8c234-2d8c-488b-9ea9-3969e9c430bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c"
Oct 30 13:45:01.038634 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:01.037519    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsr6r\" (UniqueName: \"kubernetes.io/projected/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-kube-api-access-bsr6r\") pod \"maas-default-gateway-openshift-default-67bf986f99-hbc8c\" (UID: \"d6d8c234-2d8c-488b-9ea9-3969e9c430bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c"
Oct 30 13:45:01.038634 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:01.037566    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-hbc8c\" (UID: \"d6d8c234-2d8c-488b-9ea9-3969e9c430bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c"
Oct 30 13:45:01.038634 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:01.037628    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-hbc8c\" (UID: \"d6d8c234-2d8c-488b-9ea9-3969e9c430bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c"
Oct 30 13:45:01.038634 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:01.037678    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-hbc8c\" (UID: \"d6d8c234-2d8c-488b-9ea9-3969e9c430bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c"
Oct 30 13:45:01.038634 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:01.037755    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-hbc8c\" (UID: \"d6d8c234-2d8c-488b-9ea9-3969e9c430bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c"
Oct 30 13:45:01.038634 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:01.037793    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-hbc8c\" (UID: \"d6d8c234-2d8c-488b-9ea9-3969e9c430bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c"
Oct 30 13:45:01.038634 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:01.037872    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-hbc8c\" (UID: \"d6d8c234-2d8c-488b-9ea9-3969e9c430bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c"
Oct 30 13:45:01.094891 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:45:01.094844    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38d0b166_12f0_4fab_9dd8_934f86ef2b91.slice/crio-db8cdd324cbda08cdb252ca6f471b1c823ee0fb76f7c3ec2a2f1d83ec9ddf609 WatchSource:0}: Error finding container db8cdd324cbda08cdb252ca6f471b1c823ee0fb76f7c3ec2a2f1d83ec9ddf609: Status 404 returned error can't find the container with id db8cdd324cbda08cdb252ca6f471b1c823ee0fb76f7c3ec2a2f1d83ec9ddf609
Oct 30 13:45:01.099107 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:01.099072    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889465-r98tc"]
Oct 30 13:45:01.099107 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:01.099104    5093 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 13:45:01.138987 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:01.138947    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-hbc8c\" (UID: \"d6d8c234-2d8c-488b-9ea9-3969e9c430bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c"
Oct 30 13:45:01.139159 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:01.139101    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-hbc8c\" (UID: \"d6d8c234-2d8c-488b-9ea9-3969e9c430bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c"
Oct 30 13:45:01.139242 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:01.139222    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-hbc8c\" (UID: \"d6d8c234-2d8c-488b-9ea9-3969e9c430bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c"
Oct 30 13:45:01.139310 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:01.139278    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-hbc8c\" (UID: \"d6d8c234-2d8c-488b-9ea9-3969e9c430bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c"
Oct 30 13:45:01.139357 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:01.139321    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-hbc8c\" (UID: \"d6d8c234-2d8c-488b-9ea9-3969e9c430bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c"
Oct 30 13:45:01.139394 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:01.139373    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-hbc8c\" (UID: \"d6d8c234-2d8c-488b-9ea9-3969e9c430bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c"
Oct 30 13:45:01.139512 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:01.139494    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-hbc8c\" (UID: \"d6d8c234-2d8c-488b-9ea9-3969e9c430bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c"
Oct 30 13:45:01.139559 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:01.139525    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-hbc8c\" (UID: \"d6d8c234-2d8c-488b-9ea9-3969e9c430bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c"
Oct 30 13:45:01.139559 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:01.139545    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-hbc8c\" (UID: \"d6d8c234-2d8c-488b-9ea9-3969e9c430bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c"
Oct 30 13:45:01.139641 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:01.139592    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-hbc8c\" (UID: \"d6d8c234-2d8c-488b-9ea9-3969e9c430bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c"
Oct 30 13:45:01.139676 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:01.139652    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsr6r\" (UniqueName: \"kubernetes.io/projected/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-kube-api-access-bsr6r\") pod \"maas-default-gateway-openshift-default-67bf986f99-hbc8c\" (UID: \"d6d8c234-2d8c-488b-9ea9-3969e9c430bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c"
Oct 30 13:45:01.139708 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:01.139693    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-hbc8c\" (UID: \"d6d8c234-2d8c-488b-9ea9-3969e9c430bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c"
Oct 30 13:45:01.139842 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:01.139822    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-hbc8c\" (UID: \"d6d8c234-2d8c-488b-9ea9-3969e9c430bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c"
Oct 30 13:45:01.140363 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:01.140339    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-hbc8c\" (UID: \"d6d8c234-2d8c-488b-9ea9-3969e9c430bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c"
Oct 30 13:45:01.142097 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:01.142076    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-hbc8c\" (UID: \"d6d8c234-2d8c-488b-9ea9-3969e9c430bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c"
Oct 30 13:45:01.142429 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:01.142408    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-hbc8c\" (UID: \"d6d8c234-2d8c-488b-9ea9-3969e9c430bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c"
Oct 30 13:45:01.149157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:01.149118    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889465-qtb2r"]
Oct 30 13:45:01.150287 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:45:01.150265    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6425c367_ad27_4b67_bace_4b94ff4169fc.slice/crio-fae5a05d7779207b51cb2ceb80c0e27c4260e6cccf348ff9b424d2d215545c9a WatchSource:0}: Error finding container fae5a05d7779207b51cb2ceb80c0e27c4260e6cccf348ff9b424d2d215545c9a: Status 404 returned error can't find the container with id fae5a05d7779207b51cb2ceb80c0e27c4260e6cccf348ff9b424d2d215545c9a
Oct 30 13:45:01.161266 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:01.161234    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-hbc8c\" (UID: \"d6d8c234-2d8c-488b-9ea9-3969e9c430bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c"
Oct 30 13:45:01.162762 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:01.162740    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsr6r\" (UniqueName: \"kubernetes.io/projected/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-kube-api-access-bsr6r\") pod \"maas-default-gateway-openshift-default-67bf986f99-hbc8c\" (UID: \"d6d8c234-2d8c-488b-9ea9-3969e9c430bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c"
Oct 30 13:45:01.226709 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:45:01.226665    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b6f42b5_10bf_4920_9acb_7e37f70f99e7.slice/crio-796b7acc352a7b985f39694f4004712ee095c7652621a55974d3644bda7b0592 WatchSource:0}: Error finding container 796b7acc352a7b985f39694f4004712ee095c7652621a55974d3644bda7b0592: Status 404 returned error can't find the container with id 796b7acc352a7b985f39694f4004712ee095c7652621a55974d3644bda7b0592
Oct 30 13:45:01.227518 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:01.227492    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889465-6j5w8"]
Oct 30 13:45:01.243626 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:01.243377    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c"
Oct 30 13:45:01.329498 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:01.328491    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889465-sfznh"]
Oct 30 13:45:01.333535 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:45:01.333484    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66aff1fb_d9e0_463e_81ae_71da54df6955.slice/crio-5b6f8b21b4ba8af4878bb228fcf315af2083e68d0cadb63d201ebf741ba9a7b4 WatchSource:0}: Error finding container 5b6f8b21b4ba8af4878bb228fcf315af2083e68d0cadb63d201ebf741ba9a7b4: Status 404 returned error can't find the container with id 5b6f8b21b4ba8af4878bb228fcf315af2083e68d0cadb63d201ebf741ba9a7b4
Oct 30 13:45:01.560336 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:45:01.560294    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6d8c234_2d8c_488b_9ea9_3969e9c430bf.slice/crio-63d21876a1ecd60ac896052d9953bb702d97de68cff68905956eb33ddcd2dca8 WatchSource:0}: Error finding container 63d21876a1ecd60ac896052d9953bb702d97de68cff68905956eb33ddcd2dca8: Status 404 returned error can't find the container with id 63d21876a1ecd60ac896052d9953bb702d97de68cff68905956eb33ddcd2dca8
Oct 30 13:45:01.563859 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:01.563822    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:45:01.563954 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:01.563913    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:45:01.563954 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:01.563922    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c"]
Oct 30 13:45:01.563954 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:01.563946    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:45:02.079151 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:02.079030    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889465-qtb2r" event={"ID":"6425c367-ad27-4b67-bace-4b94ff4169fc","Type":"ContainerStarted","Data":"fae5a05d7779207b51cb2ceb80c0e27c4260e6cccf348ff9b424d2d215545c9a"}
Oct 30 13:45:02.082225 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:02.082181    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889465-r98tc" event={"ID":"38d0b166-12f0-4fab-9dd8-934f86ef2b91","Type":"ContainerStarted","Data":"db8cdd324cbda08cdb252ca6f471b1c823ee0fb76f7c3ec2a2f1d83ec9ddf609"}
Oct 30 13:45:02.088300 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:02.088201    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889465-sfznh" event={"ID":"66aff1fb-d9e0-463e-81ae-71da54df6955","Type":"ContainerStarted","Data":"5b6f8b21b4ba8af4878bb228fcf315af2083e68d0cadb63d201ebf741ba9a7b4"}
Oct 30 13:45:02.092156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:02.091905    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c" event={"ID":"d6d8c234-2d8c-488b-9ea9-3969e9c430bf","Type":"ContainerStarted","Data":"87530e3ee9e3634a95376323c464547ab7e51a78d46c02c287a4c2af699a3d13"}
Oct 30 13:45:02.092156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:02.091961    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c" event={"ID":"d6d8c234-2d8c-488b-9ea9-3969e9c430bf","Type":"ContainerStarted","Data":"63d21876a1ecd60ac896052d9953bb702d97de68cff68905956eb33ddcd2dca8"}
Oct 30 13:45:02.096551 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:02.096518    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889465-6j5w8" event={"ID":"0b6f42b5-10bf-4920-9acb-7e37f70f99e7","Type":"ContainerStarted","Data":"796b7acc352a7b985f39694f4004712ee095c7652621a55974d3644bda7b0592"}
Oct 30 13:45:02.146632 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:02.146565    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c" podStartSLOduration=2.14654346 podStartE2EDuration="2.14654346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:02.142792604 +0000 UTC m=+2521.290059846" watchObservedRunningTime="2026-10-30 13:45:02.14654346 +0000 UTC m=+2521.293810701"
Oct 30 13:45:02.246394 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:02.244665    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c"
Oct 30 13:45:02.247013 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:02.246673    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hbc8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 30 13:45:02.247013 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:02.246739    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c" podUID="d6d8c234-2d8c-488b-9ea9-3969e9c430bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 30 13:45:03.275153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:03.272503    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hbc8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 30 13:45:03.275153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:03.272569    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c" podUID="d6d8c234-2d8c-488b-9ea9-3969e9c430bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 30 13:45:03.506123 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:03.505725    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd75lbz8"]
Oct 30 13:45:03.515664 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:03.515605    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd75lbz8"
Oct 30 13:45:03.579529 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:03.579444    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"payload-processing-external-model-e2e--1c277df7-dockercfg-wk62z\""
Oct 30 13:45:03.581308 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:03.581271    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"payload-processing-external-model-plugins-e2e-extmod-a-74589c\""
Oct 30 13:45:03.581452 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:03.581437    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"e2e-cfl-e402e5-provider-cred\""
Oct 30 13:45:03.588416 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:03.588387    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd75lbz8"]
Oct 30 13:45:03.620213 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:03.620147    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-6cd75lbz8\" (UID: \"b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd75lbz8"
Oct 30 13:45:03.620410 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:03.620251    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-6cd75lbz8\" (UID: \"b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd75lbz8"
Oct 30 13:45:03.620609 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:03.620536    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp8bd\" (UniqueName: \"kubernetes.io/projected/b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6-kube-api-access-kp8bd\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-6cd75lbz8\" (UID: \"b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd75lbz8"
Oct 30 13:45:03.620689 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:03.620604    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-6cd75lbz8\" (UID: \"b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd75lbz8"
Oct 30 13:45:03.670416 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:03.667469    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"routing-overlay\""
Oct 30 13:45:03.670416 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:03.667493    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"kube-root-ca.crt\""
Oct 30 13:45:03.670416 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:03.667655    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"openshift-service-ca.crt\""
Oct 30 13:45:03.723190 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:03.721792    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-6cd75lbz8\" (UID: \"b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd75lbz8"
Oct 30 13:45:03.723190 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:03.721906    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-6cd75lbz8\" (UID: \"b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd75lbz8"
Oct 30 13:45:03.723190 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:03.721949    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-6cd75lbz8\" (UID: \"b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd75lbz8"
Oct 30 13:45:03.723190 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:03.722257    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp8bd\" (UniqueName: \"kubernetes.io/projected/b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6-kube-api-access-kp8bd\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-6cd75lbz8\" (UID: \"b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd75lbz8"
Oct 30 13:45:03.723590 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:03.723300    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-6cd75lbz8\" (UID: \"b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd75lbz8"
Oct 30 13:45:03.723590 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:03.723302    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-6cd75lbz8\" (UID: \"b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd75lbz8"
Oct 30 13:45:03.763722 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:03.763119    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-6cd75lbz8\" (UID: \"b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd75lbz8"
Oct 30 13:45:03.784371 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:03.784335    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp8bd\" (UniqueName: \"kubernetes.io/projected/b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6-kube-api-access-kp8bd\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-6cd75lbz8\" (UID: \"b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd75lbz8"
Oct 30 13:45:03.892832 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:03.892389    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd75lbz8"
Oct 30 13:45:04.252323 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:04.252275    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hbc8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 30 13:45:04.252515 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:04.252361    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c" podUID="d6d8c234-2d8c-488b-9ea9-3969e9c430bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 30 13:45:04.392603 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:04.392566    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd75lbz8"]
Oct 30 13:45:05.263155 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:05.259702    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hbc8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 30 13:45:05.263155 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:05.259766    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c" podUID="d6d8c234-2d8c-488b-9ea9-3969e9c430bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 30 13:45:05.274473 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:05.274422    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd75lbz8" event={"ID":"b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6","Type":"ContainerStarted","Data":"3d0ad33cdb169176d4dec5701573c7251f10df7a5aec94d5e0bec1a9031fe2cc"}
Oct 30 13:45:05.274775 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:05.274729    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd75lbz8" event={"ID":"b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6","Type":"ContainerStarted","Data":"0fcb4b15b88849300060f2a6f0a978dd07f8936bf2942c58cb9e91bf36e464cc"}
Oct 30 13:45:05.276416 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:05.276354    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd75lbz8"
Oct 30 13:45:05.443155 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:05.436242    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd75lbz8" podStartSLOduration=2.436219942 podStartE2EDuration="2.436219942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:05.412438683 +0000 UTC m=+2524.559705938" watchObservedRunningTime="2026-10-30 13:45:05.436219942 +0000 UTC m=+2524.583487184"
Oct 30 13:45:06.256155 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:06.250323    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hbc8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 30 13:45:06.256155 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:06.250398    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c" podUID="d6d8c234-2d8c-488b-9ea9-3969e9c430bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 30 13:45:07.277158 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:07.276659    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hbc8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 30 13:45:07.277158 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:07.276727    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c" podUID="d6d8c234-2d8c-488b-9ea9-3969e9c430bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 30 13:45:08.250342 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:08.249719    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hbc8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 30 13:45:08.250342 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:08.249789    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c" podUID="d6d8c234-2d8c-488b-9ea9-3969e9c430bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 30 13:45:09.244671 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:09.244629    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hbc8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 30 13:45:09.245306 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:09.245257    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c" podUID="d6d8c234-2d8c-488b-9ea9-3969e9c430bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 30 13:45:10.248150 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:10.244496    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hbc8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 30 13:45:10.248150 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:10.244564    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c" podUID="d6d8c234-2d8c-488b-9ea9-3969e9c430bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 30 13:45:11.481440 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:11.481395    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889465-r98tc" event={"ID":"38d0b166-12f0-4fab-9dd8-934f86ef2b91","Type":"ContainerStarted","Data":"5df4f1c8eceb651528817bb9f32b5730b6834c94c7b604e31e036ea992bda78a"}
Oct 30 13:45:11.548209 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:11.538634    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889465-r98tc" podStartSLOduration=2.0705677160000002 podStartE2EDuration="11.538613828s" totalImagesPullingTime="9.468046112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:11.523271524 +0000 UTC m=+2530.670538766" watchObservedRunningTime="2026-10-30 13:45:11.538613828 +0000 UTC m=+2530.685881076"
Oct 30 13:45:11.808924 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:11.808837    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c"
Oct 30 13:45:11.809277 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:11.809253    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c"
Oct 30 13:45:11.810466 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:11.810440    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c"
Oct 30 13:45:12.492861 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:12.492828    5093 generic.go:356] "Generic (PLEG): container finished" podID="6425c367-ad27-4b67-bace-4b94ff4169fc" containerID="d46114b00693f50c21b6f3d5863e52a8727cb09e020103c46a140e08a072e71f" exitCode=60
Oct 30 13:45:12.493360 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:12.492912    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889465-qtb2r" event={"ID":"6425c367-ad27-4b67-bace-4b94ff4169fc","Type":"ContainerDied","Data":"d46114b00693f50c21b6f3d5863e52a8727cb09e020103c46a140e08a072e71f"}
Oct 30 13:45:12.493360 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:12.493288    5093 scope.go:122] "RemoveContainer" containerID="d46114b00693f50c21b6f3d5863e52a8727cb09e020103c46a140e08a072e71f"
Oct 30 13:45:12.494856 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:12.494825    5093 generic.go:356] "Generic (PLEG): container finished" podID="38d0b166-12f0-4fab-9dd8-934f86ef2b91" containerID="5df4f1c8eceb651528817bb9f32b5730b6834c94c7b604e31e036ea992bda78a" exitCode=60
Oct 30 13:45:12.494985 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:12.494891    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889465-r98tc" event={"ID":"38d0b166-12f0-4fab-9dd8-934f86ef2b91","Type":"ContainerDied","Data":"5df4f1c8eceb651528817bb9f32b5730b6834c94c7b604e31e036ea992bda78a"}
Oct 30 13:45:12.495246 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:12.495224    5093 scope.go:122] "RemoveContainer" containerID="5df4f1c8eceb651528817bb9f32b5730b6834c94c7b604e31e036ea992bda78a"
Oct 30 13:45:12.510091 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:12.510057    5093 generic.go:356] "Generic (PLEG): container finished" podID="66aff1fb-d9e0-463e-81ae-71da54df6955" containerID="2a457912bfc6245c00b3f1f07a5f179aae99238eb8306da61d3fe1800c3ad43e" exitCode=60
Oct 30 13:45:12.510594 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:12.510528    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889465-sfznh" event={"ID":"66aff1fb-d9e0-463e-81ae-71da54df6955","Type":"ContainerDied","Data":"2a457912bfc6245c00b3f1f07a5f179aae99238eb8306da61d3fe1800c3ad43e"}
Oct 30 13:45:12.511521 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:12.511498    5093 scope.go:122] "RemoveContainer" containerID="2a457912bfc6245c00b3f1f07a5f179aae99238eb8306da61d3fe1800c3ad43e"
Oct 30 13:45:12.517830 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:12.517807    5093 generic.go:356] "Generic (PLEG): container finished" podID="0b6f42b5-10bf-4920-9acb-7e37f70f99e7" containerID="592492b3d792f25555de7464ffb56577dfb4184fba7c33d2235ee0f97f7bbca6" exitCode=60
Oct 30 13:45:12.518949 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:12.518929    5093 scope.go:122] "RemoveContainer" containerID="592492b3d792f25555de7464ffb56577dfb4184fba7c33d2235ee0f97f7bbca6"
Oct 30 13:45:12.519324 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:12.519302    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889465-6j5w8" event={"ID":"0b6f42b5-10bf-4920-9acb-7e37f70f99e7","Type":"ContainerDied","Data":"592492b3d792f25555de7464ffb56577dfb4184fba7c33d2235ee0f97f7bbca6"}
Oct 30 13:45:13.541336 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:13.541290    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889465-qtb2r" event={"ID":"6425c367-ad27-4b67-bace-4b94ff4169fc","Type":"ContainerStarted","Data":"8c70e62283ec089b7ccb56aff36972ba313060d3c60bac466b2db91c6357f322"}
Oct 30 13:45:13.545178 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:13.545118    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889465-r98tc" event={"ID":"38d0b166-12f0-4fab-9dd8-934f86ef2b91","Type":"ContainerStarted","Data":"fdc207cc1cb5a7b9a11dfcd6ff67e42c0e0f6ed1affa980b5b236858fecccab6"}
Oct 30 13:45:13.548946 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:13.548911    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889465-sfznh" event={"ID":"66aff1fb-d9e0-463e-81ae-71da54df6955","Type":"ContainerStarted","Data":"5f86d689842f9e8e95f05bc7bbce21aae7531573b105fd77c0b2dcb3d60c6227"}
Oct 30 13:45:13.553584 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:13.553551    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889465-6j5w8" event={"ID":"0b6f42b5-10bf-4920-9acb-7e37f70f99e7","Type":"ContainerStarted","Data":"e9bff2415bac0c80a87ed43133bcfa4deacc512f1ebe88ca0d4b2dfdd17f69aa"}
Oct 30 13:45:13.611534 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:13.611480    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889465-qtb2r" podStartSLOduration=3.761494093 podStartE2EDuration="13.611465591s" totalImagesPullingTime="9.849971498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:13.610295921 +0000 UTC m=+2532.757563166" watchObservedRunningTime="2026-10-30 13:45:13.611465591 +0000 UTC m=+2532.758732833"
Oct 30 13:45:13.712916 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:13.712858    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889465-sfznh" podStartSLOduration=4.047926615 podStartE2EDuration="13.712842779s" totalImagesPullingTime="9.664916164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:13.710406131 +0000 UTC m=+2532.857673372" watchObservedRunningTime="2026-10-30 13:45:13.712842779 +0000 UTC m=+2532.860110020"
Oct 30 13:45:13.762180 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:13.762047    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889465-6j5w8" podStartSLOduration=3.975552925 podStartE2EDuration="13.762018322s" totalImagesPullingTime="9.786465397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:13.758284703 +0000 UTC m=+2532.905551944" watchObservedRunningTime="2026-10-30 13:45:13.762018322 +0000 UTC m=+2532.909285564"
Oct 30 13:45:14.563908 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:14.563867    5093 generic.go:356] "Generic (PLEG): container finished" podID="38d0b166-12f0-4fab-9dd8-934f86ef2b91" containerID="fdc207cc1cb5a7b9a11dfcd6ff67e42c0e0f6ed1affa980b5b236858fecccab6" exitCode=60
Oct 30 13:45:14.564454 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:14.563913    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889465-r98tc" event={"ID":"38d0b166-12f0-4fab-9dd8-934f86ef2b91","Type":"ContainerDied","Data":"fdc207cc1cb5a7b9a11dfcd6ff67e42c0e0f6ed1affa980b5b236858fecccab6"}
Oct 30 13:45:14.564454 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:14.563984    5093 scope.go:122] "RemoveContainer" containerID="5df4f1c8eceb651528817bb9f32b5730b6834c94c7b604e31e036ea992bda78a"
Oct 30 13:45:14.564454 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:14.564346    5093 scope.go:122] "RemoveContainer" containerID="fdc207cc1cb5a7b9a11dfcd6ff67e42c0e0f6ed1affa980b5b236858fecccab6"
Oct 30 13:45:14.564808 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:45:14.564774    5093 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5852ff-29889465-r98tc_odh-ai-gateway-infra(38d0b166-12f0-4fab-9dd8-934f86ef2b91)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889465-r98tc" podUID="38d0b166-12f0-4fab-9dd8-934f86ef2b91"
Oct 30 13:45:14.566605 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:14.566577    5093 generic.go:356] "Generic (PLEG): container finished" podID="66aff1fb-d9e0-463e-81ae-71da54df6955" containerID="5f86d689842f9e8e95f05bc7bbce21aae7531573b105fd77c0b2dcb3d60c6227" exitCode=60
Oct 30 13:45:14.566742 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:14.566670    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889465-sfznh" event={"ID":"66aff1fb-d9e0-463e-81ae-71da54df6955","Type":"ContainerDied","Data":"5f86d689842f9e8e95f05bc7bbce21aae7531573b105fd77c0b2dcb3d60c6227"}
Oct 30 13:45:14.567003 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:14.566986    5093 scope.go:122] "RemoveContainer" containerID="5f86d689842f9e8e95f05bc7bbce21aae7531573b105fd77c0b2dcb3d60c6227"
Oct 30 13:45:14.567466 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:45:14.567441    5093 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889465-sfznh_odh-ai-gateway-infra(66aff1fb-d9e0-463e-81ae-71da54df6955)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889465-sfznh" podUID="66aff1fb-d9e0-463e-81ae-71da54df6955"
Oct 30 13:45:14.569172 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:14.569121    5093 generic.go:356] "Generic (PLEG): container finished" podID="0b6f42b5-10bf-4920-9acb-7e37f70f99e7" containerID="e9bff2415bac0c80a87ed43133bcfa4deacc512f1ebe88ca0d4b2dfdd17f69aa" exitCode=60
Oct 30 13:45:14.569606 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:14.569585    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889465-6j5w8" event={"ID":"0b6f42b5-10bf-4920-9acb-7e37f70f99e7","Type":"ContainerDied","Data":"e9bff2415bac0c80a87ed43133bcfa4deacc512f1ebe88ca0d4b2dfdd17f69aa"}
Oct 30 13:45:14.569820 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:14.569801    5093 scope.go:122] "RemoveContainer" containerID="e9bff2415bac0c80a87ed43133bcfa4deacc512f1ebe88ca0d4b2dfdd17f69aa"
Oct 30 13:45:14.570234 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:45:14.570208    5093 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889465-6j5w8_odh-ai-gateway-infra(0b6f42b5-10bf-4920-9acb-7e37f70f99e7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889465-6j5w8" podUID="0b6f42b5-10bf-4920-9acb-7e37f70f99e7"
Oct 30 13:45:14.571838 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:14.571814    5093 generic.go:356] "Generic (PLEG): container finished" podID="6425c367-ad27-4b67-bace-4b94ff4169fc" containerID="8c70e62283ec089b7ccb56aff36972ba313060d3c60bac466b2db91c6357f322" exitCode=60
Oct 30 13:45:14.571941 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:14.571849    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889465-qtb2r" event={"ID":"6425c367-ad27-4b67-bace-4b94ff4169fc","Type":"ContainerDied","Data":"8c70e62283ec089b7ccb56aff36972ba313060d3c60bac466b2db91c6357f322"}
Oct 30 13:45:14.572243 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:14.572224    5093 scope.go:122] "RemoveContainer" containerID="8c70e62283ec089b7ccb56aff36972ba313060d3c60bac466b2db91c6357f322"
Oct 30 13:45:14.572616 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:45:14.572593    5093 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889465-qtb2r_odh-ai-gateway-infra(6425c367-ad27-4b67-bace-4b94ff4169fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889465-qtb2r" podUID="6425c367-ad27-4b67-bace-4b94ff4169fc"
Oct 30 13:45:14.579779 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:14.579759    5093 scope.go:122] "RemoveContainer" containerID="2a457912bfc6245c00b3f1f07a5f179aae99238eb8306da61d3fe1800c3ad43e"
Oct 30 13:45:14.594543 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:14.594523    5093 scope.go:122] "RemoveContainer" containerID="592492b3d792f25555de7464ffb56577dfb4184fba7c33d2235ee0f97f7bbca6"
Oct 30 13:45:14.610923 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:14.610894    5093 scope.go:122] "RemoveContainer" containerID="d46114b00693f50c21b6f3d5863e52a8727cb09e020103c46a140e08a072e71f"
Oct 30 13:45:15.013049 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:15.013012    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd75lbz8"]
Oct 30 13:45:15.017333 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:15.017299    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd75lbz8"
Oct 30 13:45:15.591520 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:15.591482    5093 scope.go:122] "RemoveContainer" containerID="8c70e62283ec089b7ccb56aff36972ba313060d3c60bac466b2db91c6357f322"
Oct 30 13:45:15.592037 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:45:15.591853    5093 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889465-qtb2r_odh-ai-gateway-infra(6425c367-ad27-4b67-bace-4b94ff4169fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889465-qtb2r" podUID="6425c367-ad27-4b67-bace-4b94ff4169fc"
Oct 30 13:45:15.593228 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:15.593203    5093 scope.go:122] "RemoveContainer" containerID="fdc207cc1cb5a7b9a11dfcd6ff67e42c0e0f6ed1affa980b5b236858fecccab6"
Oct 30 13:45:15.593723 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:45:15.593695    5093 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5852ff-29889465-r98tc_odh-ai-gateway-infra(38d0b166-12f0-4fab-9dd8-934f86ef2b91)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889465-r98tc" podUID="38d0b166-12f0-4fab-9dd8-934f86ef2b91"
Oct 30 13:45:15.595009 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:15.594987    5093 scope.go:122] "RemoveContainer" containerID="5f86d689842f9e8e95f05bc7bbce21aae7531573b105fd77c0b2dcb3d60c6227"
Oct 30 13:45:15.595303 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:45:15.595276    5093 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889465-sfznh_odh-ai-gateway-infra(66aff1fb-d9e0-463e-81ae-71da54df6955)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889465-sfznh" podUID="66aff1fb-d9e0-463e-81ae-71da54df6955"
Oct 30 13:45:15.596626 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:15.596609    5093 scope.go:122] "RemoveContainer" containerID="e9bff2415bac0c80a87ed43133bcfa4deacc512f1ebe88ca0d4b2dfdd17f69aa"
Oct 30 13:45:15.596815 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:45:15.596797    5093 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889465-6j5w8_odh-ai-gateway-infra(0b6f42b5-10bf-4920-9acb-7e37f70f99e7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889465-6j5w8" podUID="0b6f42b5-10bf-4920-9acb-7e37f70f99e7"
Oct 30 13:45:20.013683 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:20.013628    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd75lbz8" podUID="b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6" containerName="payload-processing" containerID="cri-o://3d0ad33cdb169176d4dec5701573c7251f10df7a5aec94d5e0bec1a9031fe2cc" gracePeriod=25
Oct 30 13:45:20.339019 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:20.338987    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd75lbz8"
Oct 30 13:45:20.421809 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:20.421768    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6-kube-api-access-kp8bd\" (UniqueName: \"kubernetes.io/projected/b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6-kube-api-access-kp8bd\") pod \"b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6\" (UID: \"b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6\") "
Oct 30 13:45:20.422021 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:20.421891    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6-provider-credentials\") pod \"b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6\" (UID: \"b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6\") "
Oct 30 13:45:20.422618 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:20.422540    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6-plugins-config-volume\") pod \"b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6\" (UID: \"b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6\") "
Oct 30 13:45:20.422752 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:20.422646    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6-routing-overlay-volume\") pod \"b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6\" (UID: \"b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6\") "
Oct 30 13:45:20.423366 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:20.423328    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6-routing-overlay-volume" pod "b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6" (UID: "b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:20.423491 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:20.423352    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6-plugins-config-volume" pod "b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6" (UID: "b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:20.423491 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:20.423472    5093 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6-routing-overlay-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:45:20.425487 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:20.425444    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6-kube-api-access-kp8bd" pod "b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6" (UID: "b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6"). InnerVolumeSpecName "kube-api-access-kp8bd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:20.425778 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:20.425738    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6-provider-credentials" pod "b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6" (UID: "b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:20.530662 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:20.529229    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp8bd\" (UniqueName: \"kubernetes.io/projected/b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6-kube-api-access-kp8bd\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:45:20.530662 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:20.529273    5093 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6-provider-credentials\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:45:20.530662 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:20.529289    5093 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6-plugins-config-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:45:20.692596 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:20.692552    5093 generic.go:356] "Generic (PLEG): container finished" podID="b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6" containerID="3d0ad33cdb169176d4dec5701573c7251f10df7a5aec94d5e0bec1a9031fe2cc" exitCode=0
Oct 30 13:45:20.692783 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:20.692609    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd75lbz8"
Oct 30 13:45:20.692783 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:20.692636    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd75lbz8" event={"ID":"b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6","Type":"ContainerDied","Data":"3d0ad33cdb169176d4dec5701573c7251f10df7a5aec94d5e0bec1a9031fe2cc"}
Oct 30 13:45:20.692783 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:20.692699    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd75lbz8" event={"ID":"b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6","Type":"ContainerDied","Data":"0fcb4b15b88849300060f2a6f0a978dd07f8936bf2942c58cb9e91bf36e464cc"}
Oct 30 13:45:20.692783 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:20.692732    5093 scope.go:122] "RemoveContainer" containerID="3d0ad33cdb169176d4dec5701573c7251f10df7a5aec94d5e0bec1a9031fe2cc"
Oct 30 13:45:20.704978 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:20.704957    5093 scope.go:122] "RemoveContainer" containerID="3d0ad33cdb169176d4dec5701573c7251f10df7a5aec94d5e0bec1a9031fe2cc"
Oct 30 13:45:20.705349 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:45:20.705325    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d0ad33cdb169176d4dec5701573c7251f10df7a5aec94d5e0bec1a9031fe2cc\": container with ID starting with 3d0ad33cdb169176d4dec5701573c7251f10df7a5aec94d5e0bec1a9031fe2cc not found: ID does not exist" containerID="3d0ad33cdb169176d4dec5701573c7251f10df7a5aec94d5e0bec1a9031fe2cc"
Oct 30 13:45:20.705431 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:20.705360    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d0ad33cdb169176d4dec5701573c7251f10df7a5aec94d5e0bec1a9031fe2cc"} err="failed to get container status \"3d0ad33cdb169176d4dec5701573c7251f10df7a5aec94d5e0bec1a9031fe2cc\": rpc error: code = NotFound desc = could not find container \"3d0ad33cdb169176d4dec5701573c7251f10df7a5aec94d5e0bec1a9031fe2cc\": container with ID starting with 3d0ad33cdb169176d4dec5701573c7251f10df7a5aec94d5e0bec1a9031fe2cc not found: ID does not exist"
Oct 30 13:45:20.750941 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:20.750906    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd75lbz8"]
Oct 30 13:45:20.760016 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:20.759984    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd75lbz8"]
Oct 30 13:45:21.002063 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:21.001977    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6" path="/var/lib/kubelet/pods/b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6/volumes"
Oct 30 13:45:21.181473 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:45:21.181432    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:45:23.991337 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:23.991296    5093 scope.go:122] "RemoveContainer" containerID="fdc207cc1cb5a7b9a11dfcd6ff67e42c0e0f6ed1affa980b5b236858fecccab6"
Oct 30 13:45:24.736368 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:24.736330    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889465-r98tc" event={"ID":"38d0b166-12f0-4fab-9dd8-934f86ef2b91","Type":"ContainerStarted","Data":"7b437631ed8daa0bccbea14d3fcd3a4189a72391643b63301fc9e9e50ac13ad3"}
Oct 30 13:45:24.992471 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:24.992371    5093 scope.go:122] "RemoveContainer" containerID="8c70e62283ec089b7ccb56aff36972ba313060d3c60bac466b2db91c6357f322"
Oct 30 13:45:25.748409 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:25.748363    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889465-qtb2r" event={"ID":"6425c367-ad27-4b67-bace-4b94ff4169fc","Type":"ContainerStarted","Data":"61f678b9ef49ff32fcc660aca433734c5ec1f2ec16e9df91a6bd0e5f70993994"}
Oct 30 13:45:25.750651 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:25.750618    5093 generic.go:356] "Generic (PLEG): container finished" podID="38d0b166-12f0-4fab-9dd8-934f86ef2b91" containerID="7b437631ed8daa0bccbea14d3fcd3a4189a72391643b63301fc9e9e50ac13ad3" exitCode=60
Oct 30 13:45:25.750808 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:25.750666    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889465-r98tc" event={"ID":"38d0b166-12f0-4fab-9dd8-934f86ef2b91","Type":"ContainerDied","Data":"7b437631ed8daa0bccbea14d3fcd3a4189a72391643b63301fc9e9e50ac13ad3"}
Oct 30 13:45:25.750808 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:25.750726    5093 scope.go:122] "RemoveContainer" containerID="fdc207cc1cb5a7b9a11dfcd6ff67e42c0e0f6ed1affa980b5b236858fecccab6"
Oct 30 13:45:25.751143 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:25.751113    5093 scope.go:122] "RemoveContainer" containerID="7b437631ed8daa0bccbea14d3fcd3a4189a72391643b63301fc9e9e50ac13ad3"
Oct 30 13:45:25.751569 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:45:25.751543    5093 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5852ff-29889465-r98tc_odh-ai-gateway-infra(38d0b166-12f0-4fab-9dd8-934f86ef2b91)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889465-r98tc" podUID="38d0b166-12f0-4fab-9dd8-934f86ef2b91"
Oct 30 13:45:25.798418 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:25.798382    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889465-r98tc"]
Oct 30 13:45:25.992081 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:25.992048    5093 scope.go:122] "RemoveContainer" containerID="e9bff2415bac0c80a87ed43133bcfa4deacc512f1ebe88ca0d4b2dfdd17f69aa"
Oct 30 13:45:26.763829 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:26.763732    5093 generic.go:356] "Generic (PLEG): container finished" podID="6425c367-ad27-4b67-bace-4b94ff4169fc" containerID="61f678b9ef49ff32fcc660aca433734c5ec1f2ec16e9df91a6bd0e5f70993994" exitCode=60
Oct 30 13:45:26.763829 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:26.763814    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889465-qtb2r" event={"ID":"6425c367-ad27-4b67-bace-4b94ff4169fc","Type":"ContainerDied","Data":"61f678b9ef49ff32fcc660aca433734c5ec1f2ec16e9df91a6bd0e5f70993994"}
Oct 30 13:45:26.764399 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:26.763905    5093 scope.go:122] "RemoveContainer" containerID="8c70e62283ec089b7ccb56aff36972ba313060d3c60bac466b2db91c6357f322"
Oct 30 13:45:26.764399 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:26.764378    5093 scope.go:122] "RemoveContainer" containerID="61f678b9ef49ff32fcc660aca433734c5ec1f2ec16e9df91a6bd0e5f70993994"
Oct 30 13:45:26.764851 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:45:26.764819    5093 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889465-qtb2r_odh-ai-gateway-infra(6425c367-ad27-4b67-bace-4b94ff4169fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889465-qtb2r" podUID="6425c367-ad27-4b67-bace-4b94ff4169fc"
Oct 30 13:45:26.774280 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:26.774253    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889465-6j5w8" event={"ID":"0b6f42b5-10bf-4920-9acb-7e37f70f99e7","Type":"ContainerStarted","Data":"981d307e714ea3c4a9a78f503ace6819938413da8e931d1e18d7796ccfaee137"}
Oct 30 13:45:26.831122 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:26.831076    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889465-qtb2r"]
Oct 30 13:45:26.980808 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:26.980774    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889465-r98tc"
Oct 30 13:45:27.141293 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:27.141257    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38d0b166-12f0-4fab-9dd8-934f86ef2b91-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38d0b166-12f0-4fab-9dd8-934f86ef2b91-maas-api-service-ca\") pod \"38d0b166-12f0-4fab-9dd8-934f86ef2b91\" (UID: \"38d0b166-12f0-4fab-9dd8-934f86ef2b91\") "
Oct 30 13:45:27.141566 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:27.141549    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38d0b166-12f0-4fab-9dd8-934f86ef2b91-kube-api-access-f4l9j\" (UniqueName: \"kubernetes.io/projected/38d0b166-12f0-4fab-9dd8-934f86ef2b91-kube-api-access-f4l9j\") pod \"38d0b166-12f0-4fab-9dd8-934f86ef2b91\" (UID: \"38d0b166-12f0-4fab-9dd8-934f86ef2b91\") "
Oct 30 13:45:27.141667 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:27.141640    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38d0b166-12f0-4fab-9dd8-934f86ef2b91-maas-api-service-ca" pod "38d0b166-12f0-4fab-9dd8-934f86ef2b91" (UID: "38d0b166-12f0-4fab-9dd8-934f86ef2b91"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:27.142546 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:27.142499    5093 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38d0b166-12f0-4fab-9dd8-934f86ef2b91-maas-api-service-ca\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:45:27.144573 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:27.144547    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38d0b166-12f0-4fab-9dd8-934f86ef2b91-kube-api-access-f4l9j" pod "38d0b166-12f0-4fab-9dd8-934f86ef2b91" (UID: "38d0b166-12f0-4fab-9dd8-934f86ef2b91"). InnerVolumeSpecName "kube-api-access-f4l9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:27.243942 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:27.243901    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4l9j\" (UniqueName: \"kubernetes.io/projected/38d0b166-12f0-4fab-9dd8-934f86ef2b91-kube-api-access-f4l9j\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:45:27.796568 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:27.796537    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889465-r98tc"
Oct 30 13:45:27.796989 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:27.796568    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889465-r98tc" event={"ID":"38d0b166-12f0-4fab-9dd8-934f86ef2b91","Type":"ContainerDied","Data":"db8cdd324cbda08cdb252ca6f471b1c823ee0fb76f7c3ec2a2f1d83ec9ddf609"}
Oct 30 13:45:27.796989 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:27.796669    5093 scope.go:122] "RemoveContainer" containerID="7b437631ed8daa0bccbea14d3fcd3a4189a72391643b63301fc9e9e50ac13ad3"
Oct 30 13:45:27.799587 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:27.799554    5093 generic.go:356] "Generic (PLEG): container finished" podID="0b6f42b5-10bf-4920-9acb-7e37f70f99e7" containerID="981d307e714ea3c4a9a78f503ace6819938413da8e931d1e18d7796ccfaee137" exitCode=60
Oct 30 13:45:27.799737 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:27.799622    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889465-6j5w8" event={"ID":"0b6f42b5-10bf-4920-9acb-7e37f70f99e7","Type":"ContainerDied","Data":"981d307e714ea3c4a9a78f503ace6819938413da8e931d1e18d7796ccfaee137"}
Oct 30 13:45:27.800113 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:27.800093    5093 scope.go:122] "RemoveContainer" containerID="981d307e714ea3c4a9a78f503ace6819938413da8e931d1e18d7796ccfaee137"
Oct 30 13:45:27.800536 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:45:27.800512    5093 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889465-6j5w8_odh-ai-gateway-infra(0b6f42b5-10bf-4920-9acb-7e37f70f99e7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889465-6j5w8" podUID="0b6f42b5-10bf-4920-9acb-7e37f70f99e7"
Oct 30 13:45:27.808955 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:27.808932    5093 scope.go:122] "RemoveContainer" containerID="e9bff2415bac0c80a87ed43133bcfa4deacc512f1ebe88ca0d4b2dfdd17f69aa"
Oct 30 13:45:27.931831 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:27.931784    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889465-r98tc"]
Oct 30 13:45:27.937215 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:27.937168    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889465-r98tc"]
Oct 30 13:45:27.947920 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:27.947882    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889465-6j5w8"]
Oct 30 13:45:27.992097 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:27.992063    5093 scope.go:122] "RemoveContainer" containerID="5f86d689842f9e8e95f05bc7bbce21aae7531573b105fd77c0b2dcb3d60c6227"
Oct 30 13:45:27.993821 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:27.993797    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889465-qtb2r"
Oct 30 13:45:28.161083 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:28.160986    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6425c367-ad27-4b67-bace-4b94ff4169fc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6425c367-ad27-4b67-bace-4b94ff4169fc-maas-api-service-ca\") pod \"6425c367-ad27-4b67-bace-4b94ff4169fc\" (UID: \"6425c367-ad27-4b67-bace-4b94ff4169fc\") "
Oct 30 13:45:28.161296 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:28.161110    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6425c367-ad27-4b67-bace-4b94ff4169fc-kube-api-access-76fck\" (UniqueName: \"kubernetes.io/projected/6425c367-ad27-4b67-bace-4b94ff4169fc-kube-api-access-76fck\") pod \"6425c367-ad27-4b67-bace-4b94ff4169fc\" (UID: \"6425c367-ad27-4b67-bace-4b94ff4169fc\") "
Oct 30 13:45:28.161420 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:28.161395    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6425c367-ad27-4b67-bace-4b94ff4169fc-maas-api-service-ca" pod "6425c367-ad27-4b67-bace-4b94ff4169fc" (UID: "6425c367-ad27-4b67-bace-4b94ff4169fc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:28.161871 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:28.161849    5093 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6425c367-ad27-4b67-bace-4b94ff4169fc-maas-api-service-ca\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:45:28.163438 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:28.163414    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6425c367-ad27-4b67-bace-4b94ff4169fc-kube-api-access-76fck" pod "6425c367-ad27-4b67-bace-4b94ff4169fc" (UID: "6425c367-ad27-4b67-bace-4b94ff4169fc"). InnerVolumeSpecName "kube-api-access-76fck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:28.264081 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:28.264027    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76fck\" (UniqueName: \"kubernetes.io/projected/6425c367-ad27-4b67-bace-4b94ff4169fc-kube-api-access-76fck\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:45:28.810463 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:28.810365    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889465-qtb2r" event={"ID":"6425c367-ad27-4b67-bace-4b94ff4169fc","Type":"ContainerDied","Data":"fae5a05d7779207b51cb2ceb80c0e27c4260e6cccf348ff9b424d2d215545c9a"}
Oct 30 13:45:28.810463 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:28.810437    5093 scope.go:122] "RemoveContainer" containerID="61f678b9ef49ff32fcc660aca433734c5ec1f2ec16e9df91a6bd0e5f70993994"
Oct 30 13:45:28.810997 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:28.810377    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889465-qtb2r"
Oct 30 13:45:28.813954 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:28.813923    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889465-sfznh" event={"ID":"66aff1fb-d9e0-463e-81ae-71da54df6955","Type":"ContainerStarted","Data":"2ba2640db986ab4853418966d23eed7e5880af6c68ed5e5deb6b9af0fa816adc"}
Oct 30 13:45:28.925822 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:28.925783    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889465-qtb2r"]
Oct 30 13:45:28.932581 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:28.932547    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889465-qtb2r"]
Oct 30 13:45:29.000576 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:29.000543    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889465-6j5w8"
Oct 30 13:45:29.003029 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:29.002999    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38d0b166-12f0-4fab-9dd8-934f86ef2b91" path="/var/lib/kubelet/pods/38d0b166-12f0-4fab-9dd8-934f86ef2b91/volumes"
Oct 30 13:45:29.003581 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:29.003560    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6425c367-ad27-4b67-bace-4b94ff4169fc" path="/var/lib/kubelet/pods/6425c367-ad27-4b67-bace-4b94ff4169fc/volumes"
Oct 30 13:45:29.076598 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:29.076498    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b6f42b5-10bf-4920-9acb-7e37f70f99e7-kube-api-access-vd6n7\" (UniqueName: \"kubernetes.io/projected/0b6f42b5-10bf-4920-9acb-7e37f70f99e7-kube-api-access-vd6n7\") pod \"0b6f42b5-10bf-4920-9acb-7e37f70f99e7\" (UID: \"0b6f42b5-10bf-4920-9acb-7e37f70f99e7\") "
Oct 30 13:45:29.076756 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:29.076638    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b6f42b5-10bf-4920-9acb-7e37f70f99e7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b6f42b5-10bf-4920-9acb-7e37f70f99e7-maas-api-service-ca\") pod \"0b6f42b5-10bf-4920-9acb-7e37f70f99e7\" (UID: \"0b6f42b5-10bf-4920-9acb-7e37f70f99e7\") "
Oct 30 13:45:29.077246 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:29.077217    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b6f42b5-10bf-4920-9acb-7e37f70f99e7-maas-api-service-ca" pod "0b6f42b5-10bf-4920-9acb-7e37f70f99e7" (UID: "0b6f42b5-10bf-4920-9acb-7e37f70f99e7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:29.077442 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:29.077416    5093 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b6f42b5-10bf-4920-9acb-7e37f70f99e7-maas-api-service-ca\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:45:29.079357 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:29.079329    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b6f42b5-10bf-4920-9acb-7e37f70f99e7-kube-api-access-vd6n7" pod "0b6f42b5-10bf-4920-9acb-7e37f70f99e7" (UID: "0b6f42b5-10bf-4920-9acb-7e37f70f99e7"). InnerVolumeSpecName "kube-api-access-vd6n7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:29.179406 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:29.179366    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd6n7\" (UniqueName: \"kubernetes.io/projected/0b6f42b5-10bf-4920-9acb-7e37f70f99e7-kube-api-access-vd6n7\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:45:29.828036 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:29.827999    5093 generic.go:356] "Generic (PLEG): container finished" podID="66aff1fb-d9e0-463e-81ae-71da54df6955" containerID="2ba2640db986ab4853418966d23eed7e5880af6c68ed5e5deb6b9af0fa816adc" exitCode=60
Oct 30 13:45:29.828539 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:29.828044    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889465-sfznh" event={"ID":"66aff1fb-d9e0-463e-81ae-71da54df6955","Type":"ContainerDied","Data":"2ba2640db986ab4853418966d23eed7e5880af6c68ed5e5deb6b9af0fa816adc"}
Oct 30 13:45:29.828539 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:29.828124    5093 scope.go:122] "RemoveContainer" containerID="5f86d689842f9e8e95f05bc7bbce21aae7531573b105fd77c0b2dcb3d60c6227"
Oct 30 13:45:29.828643 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:29.828619    5093 scope.go:122] "RemoveContainer" containerID="2ba2640db986ab4853418966d23eed7e5880af6c68ed5e5deb6b9af0fa816adc"
Oct 30 13:45:29.829080 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:45:29.829053    5093 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889465-sfznh_odh-ai-gateway-infra(66aff1fb-d9e0-463e-81ae-71da54df6955)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889465-sfznh" podUID="66aff1fb-d9e0-463e-81ae-71da54df6955"
Oct 30 13:45:29.829903 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:29.829871    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889465-6j5w8"
Oct 30 13:45:29.830030 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:29.829879    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889465-6j5w8" event={"ID":"0b6f42b5-10bf-4920-9acb-7e37f70f99e7","Type":"ContainerDied","Data":"796b7acc352a7b985f39694f4004712ee095c7652621a55974d3644bda7b0592"}
Oct 30 13:45:29.843802 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:29.843775    5093 scope.go:122] "RemoveContainer" containerID="981d307e714ea3c4a9a78f503ace6819938413da8e931d1e18d7796ccfaee137"
Oct 30 13:45:29.890913 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:29.890875    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889465-sfznh"]
Oct 30 13:45:29.922995 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:29.922959    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889465-6j5w8"]
Oct 30 13:45:29.936731 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:29.936696    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889465-6j5w8"]
Oct 30 13:45:31.007208 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:31.007169    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b6f42b5-10bf-4920-9acb-7e37f70f99e7" path="/var/lib/kubelet/pods/0b6f42b5-10bf-4920-9acb-7e37f70f99e7/volumes"
Oct 30 13:45:31.051349 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:31.051313    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889465-sfznh"
Oct 30 13:45:31.208196 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:31.208118    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66aff1fb-d9e0-463e-81ae-71da54df6955-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66aff1fb-d9e0-463e-81ae-71da54df6955-maas-api-service-ca\") pod \"66aff1fb-d9e0-463e-81ae-71da54df6955\" (UID: \"66aff1fb-d9e0-463e-81ae-71da54df6955\") "
Oct 30 13:45:31.208356 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:31.208224    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66aff1fb-d9e0-463e-81ae-71da54df6955-kube-api-access-px9vq\" (UniqueName: \"kubernetes.io/projected/66aff1fb-d9e0-463e-81ae-71da54df6955-kube-api-access-px9vq\") pod \"66aff1fb-d9e0-463e-81ae-71da54df6955\" (UID: \"66aff1fb-d9e0-463e-81ae-71da54df6955\") "
Oct 30 13:45:31.208524 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:31.208500    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66aff1fb-d9e0-463e-81ae-71da54df6955-maas-api-service-ca" pod "66aff1fb-d9e0-463e-81ae-71da54df6955" (UID: "66aff1fb-d9e0-463e-81ae-71da54df6955"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:31.208976 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:31.208954    5093 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66aff1fb-d9e0-463e-81ae-71da54df6955-maas-api-service-ca\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:45:31.210570 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:31.210548    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66aff1fb-d9e0-463e-81ae-71da54df6955-kube-api-access-px9vq" pod "66aff1fb-d9e0-463e-81ae-71da54df6955" (UID: "66aff1fb-d9e0-463e-81ae-71da54df6955"). InnerVolumeSpecName "kube-api-access-px9vq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:31.310103 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:31.310062    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px9vq\" (UniqueName: \"kubernetes.io/projected/66aff1fb-d9e0-463e-81ae-71da54df6955-kube-api-access-px9vq\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:45:31.854930 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:31.854897    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889465-sfznh"
Oct 30 13:45:31.855207 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:31.854902    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889465-sfznh" event={"ID":"66aff1fb-d9e0-463e-81ae-71da54df6955","Type":"ContainerDied","Data":"5b6f8b21b4ba8af4878bb228fcf315af2083e68d0cadb63d201ebf741ba9a7b4"}
Oct 30 13:45:31.855207 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:31.855059    5093 scope.go:122] "RemoveContainer" containerID="2ba2640db986ab4853418966d23eed7e5880af6c68ed5e5deb6b9af0fa816adc"
Oct 30 13:45:31.939781 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:31.939746    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889465-sfznh"]
Oct 30 13:45:31.947239 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:31.947207    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889465-sfznh"]
Oct 30 13:45:33.000466 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:33.000421    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66aff1fb-d9e0-463e-81ae-71da54df6955" path="/var/lib/kubelet/pods/66aff1fb-d9e0-463e-81ae-71da54df6955/volumes"
Oct 30 13:45:37.026726 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.026678    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7qmdwx"]
Oct 30 13:45:37.027737 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.027711    5093 cpu_manager.go:414] "containerMap: removing container" podUID="b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6" containerName="payload-processing"
Oct 30 13:45:37.027737 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.027737    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6" containerName="payload-processing"
Oct 30 13:45:37.027914 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.027761    5093 cpu_manager.go:414] "containerMap: removing container" podUID="38d0b166-12f0-4fab-9dd8-934f86ef2b91" containerName="cleanup"
Oct 30 13:45:37.027914 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.027769    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38d0b166-12f0-4fab-9dd8-934f86ef2b91" containerName="cleanup"
Oct 30 13:45:37.027914 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.027778    5093 cpu_manager.go:414] "containerMap: removing container" podUID="6425c367-ad27-4b67-bace-4b94ff4169fc" containerName="cleanup"
Oct 30 13:45:37.027914 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.027785    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6425c367-ad27-4b67-bace-4b94ff4169fc" containerName="cleanup"
Oct 30 13:45:37.027914 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.027798    5093 cpu_manager.go:414] "containerMap: removing container" podUID="0b6f42b5-10bf-4920-9acb-7e37f70f99e7" containerName="cleanup"
Oct 30 13:45:37.027914 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.027805    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b6f42b5-10bf-4920-9acb-7e37f70f99e7" containerName="cleanup"
Oct 30 13:45:37.027914 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.027830    5093 cpu_manager.go:414] "containerMap: removing container" podUID="38d0b166-12f0-4fab-9dd8-934f86ef2b91" containerName="cleanup"
Oct 30 13:45:37.027914 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.027839    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38d0b166-12f0-4fab-9dd8-934f86ef2b91" containerName="cleanup"
Oct 30 13:45:37.027914 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.027852    5093 cpu_manager.go:414] "containerMap: removing container" podUID="6425c367-ad27-4b67-bace-4b94ff4169fc" containerName="cleanup"
Oct 30 13:45:37.027914 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.027859    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6425c367-ad27-4b67-bace-4b94ff4169fc" containerName="cleanup"
Oct 30 13:45:37.027914 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.027885    5093 cpu_manager.go:414] "containerMap: removing container" podUID="0b6f42b5-10bf-4920-9acb-7e37f70f99e7" containerName="cleanup"
Oct 30 13:45:37.027914 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.027893    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b6f42b5-10bf-4920-9acb-7e37f70f99e7" containerName="cleanup"
Oct 30 13:45:37.028592 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.027919    5093 cpu_manager.go:414] "containerMap: removing container" podUID="0b6f42b5-10bf-4920-9acb-7e37f70f99e7" containerName="cleanup"
Oct 30 13:45:37.028592 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.027927    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b6f42b5-10bf-4920-9acb-7e37f70f99e7" containerName="cleanup"
Oct 30 13:45:37.028592 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.027955    5093 cpu_manager.go:414] "containerMap: removing container" podUID="66aff1fb-d9e0-463e-81ae-71da54df6955" containerName="cleanup"
Oct 30 13:45:37.028592 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.027964    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66aff1fb-d9e0-463e-81ae-71da54df6955" containerName="cleanup"
Oct 30 13:45:37.028592 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.027977    5093 cpu_manager.go:414] "containerMap: removing container" podUID="66aff1fb-d9e0-463e-81ae-71da54df6955" containerName="cleanup"
Oct 30 13:45:37.028592 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.027984    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66aff1fb-d9e0-463e-81ae-71da54df6955" containerName="cleanup"
Oct 30 13:45:37.028592 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.028016    5093 cpu_manager.go:414] "containerMap: removing container" podUID="66aff1fb-d9e0-463e-81ae-71da54df6955" containerName="cleanup"
Oct 30 13:45:37.028592 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.028025    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66aff1fb-d9e0-463e-81ae-71da54df6955" containerName="cleanup"
Oct 30 13:45:37.028592 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.028238    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="38d0b166-12f0-4fab-9dd8-934f86ef2b91" containerName="cleanup"
Oct 30 13:45:37.028592 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.028255    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="6425c367-ad27-4b67-bace-4b94ff4169fc" containerName="cleanup"
Oct 30 13:45:37.028592 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.028267    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b6f42b5-10bf-4920-9acb-7e37f70f99e7" containerName="cleanup"
Oct 30 13:45:37.028592 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.028284    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="66aff1fb-d9e0-463e-81ae-71da54df6955" containerName="cleanup"
Oct 30 13:45:37.028592 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.028294    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="66aff1fb-d9e0-463e-81ae-71da54df6955" containerName="cleanup"
Oct 30 13:45:37.028592 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.028308    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1f0db0e-8c9d-4636-9073-7e0d3c2f46f6" containerName="payload-processing"
Oct 30 13:45:37.028592 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.028323    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="38d0b166-12f0-4fab-9dd8-934f86ef2b91" containerName="cleanup"
Oct 30 13:45:37.028592 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.028330    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="6425c367-ad27-4b67-bace-4b94ff4169fc" containerName="cleanup"
Oct 30 13:45:37.028592 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.028339    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b6f42b5-10bf-4920-9acb-7e37f70f99e7" containerName="cleanup"
Oct 30 13:45:37.034492 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.034467    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7qmdwx"
Oct 30 13:45:37.063576 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.063541    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7qmdwx"]
Oct 30 13:45:37.072038 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.072003    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"payload-processing-external-model-e2e--1c277df7-dockercfg-59xr6\""
Oct 30 13:45:37.072417 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.072078    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"agc-cred-f06454846ef2\""
Oct 30 13:45:37.072540 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.072220    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"payload-processing-external-model-plugins-e2e-extmod-a-74589c\""
Oct 30 13:45:37.072540 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.072294    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"routing-overlay\""
Oct 30 13:45:37.072540 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.072352    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"kube-root-ca.crt\""
Oct 30 13:45:37.072540 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.072356    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"openshift-service-ca.crt\""
Oct 30 13:45:37.111919 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.111885    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a247c1fd-0e91-4de7-aba7-465667dbd736-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-76b7qmdwx\" (UID: \"a247c1fd-0e91-4de7-aba7-465667dbd736\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7qmdwx"
Oct 30 13:45:37.112112 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.111950    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a247c1fd-0e91-4de7-aba7-465667dbd736-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-76b7qmdwx\" (UID: \"a247c1fd-0e91-4de7-aba7-465667dbd736\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7qmdwx"
Oct 30 13:45:37.112225 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.112113    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nhxf\" (UniqueName: \"kubernetes.io/projected/a247c1fd-0e91-4de7-aba7-465667dbd736-kube-api-access-7nhxf\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-76b7qmdwx\" (UID: \"a247c1fd-0e91-4de7-aba7-465667dbd736\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7qmdwx"
Oct 30 13:45:37.112282 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.112226    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a247c1fd-0e91-4de7-aba7-465667dbd736-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-76b7qmdwx\" (UID: \"a247c1fd-0e91-4de7-aba7-465667dbd736\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7qmdwx"
Oct 30 13:45:37.231153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.226167    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a247c1fd-0e91-4de7-aba7-465667dbd736-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-76b7qmdwx\" (UID: \"a247c1fd-0e91-4de7-aba7-465667dbd736\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7qmdwx"
Oct 30 13:45:37.231153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.226222    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a247c1fd-0e91-4de7-aba7-465667dbd736-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-76b7qmdwx\" (UID: \"a247c1fd-0e91-4de7-aba7-465667dbd736\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7qmdwx"
Oct 30 13:45:37.231153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.226285    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nhxf\" (UniqueName: \"kubernetes.io/projected/a247c1fd-0e91-4de7-aba7-465667dbd736-kube-api-access-7nhxf\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-76b7qmdwx\" (UID: \"a247c1fd-0e91-4de7-aba7-465667dbd736\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7qmdwx"
Oct 30 13:45:37.231153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.226338    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a247c1fd-0e91-4de7-aba7-465667dbd736-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-76b7qmdwx\" (UID: \"a247c1fd-0e91-4de7-aba7-465667dbd736\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7qmdwx"
Oct 30 13:45:37.231153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.227520    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a247c1fd-0e91-4de7-aba7-465667dbd736-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-76b7qmdwx\" (UID: \"a247c1fd-0e91-4de7-aba7-465667dbd736\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7qmdwx"
Oct 30 13:45:37.231153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.228102    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a247c1fd-0e91-4de7-aba7-465667dbd736-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-76b7qmdwx\" (UID: \"a247c1fd-0e91-4de7-aba7-465667dbd736\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7qmdwx"
Oct 30 13:45:37.266801 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.256206    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a247c1fd-0e91-4de7-aba7-465667dbd736-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-76b7qmdwx\" (UID: \"a247c1fd-0e91-4de7-aba7-465667dbd736\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7qmdwx"
Oct 30 13:45:37.290397 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.283405    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nhxf\" (UniqueName: \"kubernetes.io/projected/a247c1fd-0e91-4de7-aba7-465667dbd736-kube-api-access-7nhxf\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-76b7qmdwx\" (UID: \"a247c1fd-0e91-4de7-aba7-465667dbd736\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7qmdwx"
Oct 30 13:45:37.350367 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.350316    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7qmdwx"
Oct 30 13:45:37.689882 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:45:37.689839    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda247c1fd_0e91_4de7_aba7_465667dbd736.slice/crio-5eedb95b03deceedf36ea1dd99ae8e68703b266f9c4ccc489ef8c58f86f1e2a4 WatchSource:0}: Error finding container 5eedb95b03deceedf36ea1dd99ae8e68703b266f9c4ccc489ef8c58f86f1e2a4: Status 404 returned error can't find the container with id 5eedb95b03deceedf36ea1dd99ae8e68703b266f9c4ccc489ef8c58f86f1e2a4
Oct 30 13:45:37.690356 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.690329    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7qmdwx"]
Oct 30 13:45:37.966408 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.966297    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7qmdwx" event={"ID":"a247c1fd-0e91-4de7-aba7-465667dbd736","Type":"ContainerStarted","Data":"a1c65fb6086ffb22767385d2d1932dbd0eb29fd1f40fd111ff43e48e5ebb5464"}
Oct 30 13:45:37.966408 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.966369    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7qmdwx" event={"ID":"a247c1fd-0e91-4de7-aba7-465667dbd736","Type":"ContainerStarted","Data":"5eedb95b03deceedf36ea1dd99ae8e68703b266f9c4ccc489ef8c58f86f1e2a4"}
Oct 30 13:45:37.966712 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:37.966679    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7qmdwx"
Oct 30 13:45:38.014561 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:38.014499    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7qmdwx" podStartSLOduration=2.014480586 podStartE2EDuration="2.014480586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:38.0124953 +0000 UTC m=+2557.159762541" watchObservedRunningTime="2026-10-30 13:45:38.014480586 +0000 UTC m=+2557.161747848"
Oct 30 13:45:48.980067 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:45:48.979973    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7qmdwx"
Oct 30 13:45:51.168850 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:45:51.168805    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:46:13.912106 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:13.912067    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn"]
Oct 30 13:46:13.912681 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:13.912293    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn" podUID="de03af69-2ff9-4add-a940-740935198076" containerName="istio-proxy" containerID="cri-o://10a47553747aa9b3ccf7bf837fbdaaeeeeb1935e4daa11719b3564f6782ef83a" gracePeriod=30
Oct 30 13:46:18.626411 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:18.626365    5093 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:46:18.626816 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:18.626425    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn" podUID="de03af69-2ff9-4add-a940-740935198076" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:46:19.295902 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:19.295864    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn"
Oct 30 13:46:19.406242 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:19.397341    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de03af69-2ff9-4add-a940-740935198076-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de03af69-2ff9-4add-a940-740935198076-workload-certs\") pod \"de03af69-2ff9-4add-a940-740935198076\" (UID: \"de03af69-2ff9-4add-a940-740935198076\") "
Oct 30 13:46:19.406242 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:19.397457    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de03af69-2ff9-4add-a940-740935198076-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de03af69-2ff9-4add-a940-740935198076-workload-socket\") pod \"de03af69-2ff9-4add-a940-740935198076\" (UID: \"de03af69-2ff9-4add-a940-740935198076\") "
Oct 30 13:46:19.406242 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:19.397542    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de03af69-2ff9-4add-a940-740935198076-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de03af69-2ff9-4add-a940-740935198076-istio-data\") pod \"de03af69-2ff9-4add-a940-740935198076\" (UID: \"de03af69-2ff9-4add-a940-740935198076\") "
Oct 30 13:46:19.406242 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:19.397603    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de03af69-2ff9-4add-a940-740935198076-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de03af69-2ff9-4add-a940-740935198076-istio-envoy\") pod \"de03af69-2ff9-4add-a940-740935198076\" (UID: \"de03af69-2ff9-4add-a940-740935198076\") "
Oct 30 13:46:19.406242 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:19.397640    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de03af69-2ff9-4add-a940-740935198076-istio-token\" (UniqueName: \"kubernetes.io/projected/de03af69-2ff9-4add-a940-740935198076-istio-token\") pod \"de03af69-2ff9-4add-a940-740935198076\" (UID: \"de03af69-2ff9-4add-a940-740935198076\") "
Oct 30 13:46:19.406242 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:19.397739    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de03af69-2ff9-4add-a940-740935198076-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de03af69-2ff9-4add-a940-740935198076-credential-socket\") pod \"de03af69-2ff9-4add-a940-740935198076\" (UID: \"de03af69-2ff9-4add-a940-740935198076\") "
Oct 30 13:46:19.406242 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:19.397786    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de03af69-2ff9-4add-a940-740935198076-kube-api-access-kwvph\" (UniqueName: \"kubernetes.io/projected/de03af69-2ff9-4add-a940-740935198076-kube-api-access-kwvph\") pod \"de03af69-2ff9-4add-a940-740935198076\" (UID: \"de03af69-2ff9-4add-a940-740935198076\") "
Oct 30 13:46:19.406242 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:19.397901    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de03af69-2ff9-4add-a940-740935198076-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de03af69-2ff9-4add-a940-740935198076-istiod-ca-cert\") pod \"de03af69-2ff9-4add-a940-740935198076\" (UID: \"de03af69-2ff9-4add-a940-740935198076\") "
Oct 30 13:46:19.406242 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:19.397972    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de03af69-2ff9-4add-a940-740935198076-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de03af69-2ff9-4add-a940-740935198076-istio-podinfo\") pod \"de03af69-2ff9-4add-a940-740935198076\" (UID: \"de03af69-2ff9-4add-a940-740935198076\") "
Oct 30 13:46:19.406242 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:19.402226    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de03af69-2ff9-4add-a940-740935198076-workload-socket" pod "de03af69-2ff9-4add-a940-740935198076" (UID: "de03af69-2ff9-4add-a940-740935198076"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:46:19.406242 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:19.403297    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de03af69-2ff9-4add-a940-740935198076-workload-certs" pod "de03af69-2ff9-4add-a940-740935198076" (UID: "de03af69-2ff9-4add-a940-740935198076"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:46:19.406242 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:19.403943    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de03af69-2ff9-4add-a940-740935198076-credential-socket" pod "de03af69-2ff9-4add-a940-740935198076" (UID: "de03af69-2ff9-4add-a940-740935198076"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:46:19.406242 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:19.404472    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de03af69-2ff9-4add-a940-740935198076-istiod-ca-cert" pod "de03af69-2ff9-4add-a940-740935198076" (UID: "de03af69-2ff9-4add-a940-740935198076"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:46:19.413511 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:19.413478    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de03af69-2ff9-4add-a940-740935198076-istio-data" pod "de03af69-2ff9-4add-a940-740935198076" (UID: "de03af69-2ff9-4add-a940-740935198076"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:46:19.442827 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:19.442722    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de03af69-2ff9-4add-a940-740935198076-kube-api-access-kwvph" pod "de03af69-2ff9-4add-a940-740935198076" (UID: "de03af69-2ff9-4add-a940-740935198076"). InnerVolumeSpecName "kube-api-access-kwvph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:46:19.443624 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:19.443586    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de03af69-2ff9-4add-a940-740935198076-istio-podinfo" pod "de03af69-2ff9-4add-a940-740935198076" (UID: "de03af69-2ff9-4add-a940-740935198076"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:46:19.451662 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:19.451616    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de03af69-2ff9-4add-a940-740935198076-istio-envoy" pod "de03af69-2ff9-4add-a940-740935198076" (UID: "de03af69-2ff9-4add-a940-740935198076"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:46:19.482630 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:19.482317    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de03af69-2ff9-4add-a940-740935198076-istio-token" pod "de03af69-2ff9-4add-a940-740935198076" (UID: "de03af69-2ff9-4add-a940-740935198076"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:46:19.507253 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:19.503190    5093 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de03af69-2ff9-4add-a940-740935198076-istio-podinfo\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:46:19.507253 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:19.503231    5093 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de03af69-2ff9-4add-a940-740935198076-workload-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:46:19.507253 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:19.503246    5093 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de03af69-2ff9-4add-a940-740935198076-workload-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:46:19.507253 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:19.503259    5093 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de03af69-2ff9-4add-a940-740935198076-istio-data\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:46:19.507253 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:19.503272    5093 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de03af69-2ff9-4add-a940-740935198076-istio-envoy\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:46:19.507253 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:19.503284    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de03af69-2ff9-4add-a940-740935198076-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:46:19.507253 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:19.503297    5093 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de03af69-2ff9-4add-a940-740935198076-credential-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:46:19.507253 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:19.503313    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwvph\" (UniqueName: \"kubernetes.io/projected/de03af69-2ff9-4add-a940-740935198076-kube-api-access-kwvph\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:46:19.507253 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:19.503328    5093 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de03af69-2ff9-4add-a940-740935198076-istiod-ca-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:46:19.510100 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:19.509997    5093 generic.go:356] "Generic (PLEG): container finished" podID="de03af69-2ff9-4add-a940-740935198076" containerID="10a47553747aa9b3ccf7bf837fbdaaeeeeb1935e4daa11719b3564f6782ef83a" exitCode=0
Oct 30 13:46:19.510100 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:19.510074    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn"
Oct 30 13:46:19.510319 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:19.510196    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn" event={"ID":"de03af69-2ff9-4add-a940-740935198076","Type":"ContainerDied","Data":"10a47553747aa9b3ccf7bf837fbdaaeeeeb1935e4daa11719b3564f6782ef83a"}
Oct 30 13:46:19.510319 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:19.510245    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn" event={"ID":"de03af69-2ff9-4add-a940-740935198076","Type":"ContainerDied","Data":"1e4f6faed3c5e30e4d0c2a068b0c5829786d7b817fcb7a48e0a583703835f058"}
Oct 30 13:46:19.510319 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:19.510276    5093 scope.go:122] "RemoveContainer" containerID="10a47553747aa9b3ccf7bf837fbdaaeeeeb1935e4daa11719b3564f6782ef83a"
Oct 30 13:46:19.564006 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:19.563974    5093 scope.go:122] "RemoveContainer" containerID="10a47553747aa9b3ccf7bf837fbdaaeeeeb1935e4daa11719b3564f6782ef83a"
Oct 30 13:46:19.564490 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:46:19.564456    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10a47553747aa9b3ccf7bf837fbdaaeeeeb1935e4daa11719b3564f6782ef83a\": container with ID starting with 10a47553747aa9b3ccf7bf837fbdaaeeeeb1935e4daa11719b3564f6782ef83a not found: ID does not exist" containerID="10a47553747aa9b3ccf7bf837fbdaaeeeeb1935e4daa11719b3564f6782ef83a"
Oct 30 13:46:19.564575 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:19.564503    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10a47553747aa9b3ccf7bf837fbdaaeeeeb1935e4daa11719b3564f6782ef83a"} err="failed to get container status \"10a47553747aa9b3ccf7bf837fbdaaeeeeb1935e4daa11719b3564f6782ef83a\": rpc error: code = NotFound desc = could not find container \"10a47553747aa9b3ccf7bf837fbdaaeeeeb1935e4daa11719b3564f6782ef83a\": container with ID starting with 10a47553747aa9b3ccf7bf837fbdaaeeeeb1935e4daa11719b3564f6782ef83a not found: ID does not exist"
Oct 30 13:46:19.598200 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:19.598162    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn"]
Oct 30 13:46:19.606734 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:19.606695    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-lv2pn"]
Oct 30 13:46:21.002575 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:21.002540    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de03af69-2ff9-4add-a940-740935198076" path="/var/lib/kubelet/pods/de03af69-2ff9-4add-a940-740935198076/volumes"
Oct 30 13:46:21.168816 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:46:21.168778    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:46:31.035041 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:31.035001    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6cd7tcwgw"]
Oct 30 13:46:31.036030 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:31.036002    5093 cpu_manager.go:414] "containerMap: removing container" podUID="38d0b166-12f0-4fab-9dd8-934f86ef2b91" containerName="cleanup"
Oct 30 13:46:31.036178 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:31.036031    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38d0b166-12f0-4fab-9dd8-934f86ef2b91" containerName="cleanup"
Oct 30 13:46:31.036178 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:31.036063    5093 cpu_manager.go:414] "containerMap: removing container" podUID="6425c367-ad27-4b67-bace-4b94ff4169fc" containerName="cleanup"
Oct 30 13:46:31.036178 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:31.036072    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6425c367-ad27-4b67-bace-4b94ff4169fc" containerName="cleanup"
Oct 30 13:46:31.036178 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:31.036118    5093 cpu_manager.go:414] "containerMap: removing container" podUID="de03af69-2ff9-4add-a940-740935198076" containerName="istio-proxy"
Oct 30 13:46:31.036178 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:31.036141    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de03af69-2ff9-4add-a940-740935198076" containerName="istio-proxy"
Oct 30 13:46:31.036443 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:31.036334    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="38d0b166-12f0-4fab-9dd8-934f86ef2b91" containerName="cleanup"
Oct 30 13:46:31.036443 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:31.036356    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="6425c367-ad27-4b67-bace-4b94ff4169fc" containerName="cleanup"
Oct 30 13:46:31.036443 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:31.036370    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b6f42b5-10bf-4920-9acb-7e37f70f99e7" containerName="cleanup"
Oct 30 13:46:31.036443 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:31.036384    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="de03af69-2ff9-4add-a940-740935198076" containerName="istio-proxy"
Oct 30 13:46:31.036443 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:31.036397    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="66aff1fb-d9e0-463e-81ae-71da54df6955" containerName="cleanup"
Oct 30 13:46:31.040465 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:31.040440    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6cd7tcwgw"
Oct 30 13:46:31.059567 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:31.059536    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6cd7tcwgw"]
Oct 30 13:46:31.066721 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:31.066440    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-310546\"/\"payload-processing-external-model-plugins-e2e-extmod-a-310546\""
Oct 30 13:46:31.066721 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:31.066571    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-310546\"/\"routing-overlay\""
Oct 30 13:46:31.066721 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:31.066660    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-310546\"/\"kube-root-ca.crt\""
Oct 30 13:46:31.067031 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:31.066796    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-310546\"/\"agc-cred-80995a0f88ba\""
Oct 30 13:46:31.067031 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:31.066806    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-310546\"/\"openshift-service-ca.crt\""
Oct 30 13:46:31.075193 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:31.071355    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-310546\"/\"payload-processing-external-model-e2e--175bf124-dockercfg-k2xvk\""
Oct 30 13:46:31.136366 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:31.136260    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6f6422f2-f1b6-433f-93a3-52edd828c682-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-310546-6cd7tcwgw\" (UID: \"6f6422f2-f1b6-433f-93a3-52edd828c682\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6cd7tcwgw"
Oct 30 13:46:31.136770 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:31.136746    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f6422f2-f1b6-433f-93a3-52edd828c682-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-310546-6cd7tcwgw\" (UID: \"6f6422f2-f1b6-433f-93a3-52edd828c682\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6cd7tcwgw"
Oct 30 13:46:31.137096 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:31.137076    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6f6422f2-f1b6-433f-93a3-52edd828c682-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-310546-6cd7tcwgw\" (UID: \"6f6422f2-f1b6-433f-93a3-52edd828c682\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6cd7tcwgw"
Oct 30 13:46:31.137779 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:31.137759    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrwws\" (UniqueName: \"kubernetes.io/projected/6f6422f2-f1b6-433f-93a3-52edd828c682-kube-api-access-vrwws\") pod \"payload-processing-external-model-e2e-extmod-a-310546-6cd7tcwgw\" (UID: \"6f6422f2-f1b6-433f-93a3-52edd828c682\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6cd7tcwgw"
Oct 30 13:46:31.257330 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:31.257282    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6f6422f2-f1b6-433f-93a3-52edd828c682-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-310546-6cd7tcwgw\" (UID: \"6f6422f2-f1b6-433f-93a3-52edd828c682\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6cd7tcwgw"
Oct 30 13:46:31.258283 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:31.258244    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrwws\" (UniqueName: \"kubernetes.io/projected/6f6422f2-f1b6-433f-93a3-52edd828c682-kube-api-access-vrwws\") pod \"payload-processing-external-model-e2e-extmod-a-310546-6cd7tcwgw\" (UID: \"6f6422f2-f1b6-433f-93a3-52edd828c682\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6cd7tcwgw"
Oct 30 13:46:31.262184 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:31.260926    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6f6422f2-f1b6-433f-93a3-52edd828c682-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-310546-6cd7tcwgw\" (UID: \"6f6422f2-f1b6-433f-93a3-52edd828c682\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6cd7tcwgw"
Oct 30 13:46:31.262184 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:31.261050    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f6422f2-f1b6-433f-93a3-52edd828c682-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-310546-6cd7tcwgw\" (UID: \"6f6422f2-f1b6-433f-93a3-52edd828c682\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6cd7tcwgw"
Oct 30 13:46:31.262184 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:31.261893    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6f6422f2-f1b6-433f-93a3-52edd828c682-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-310546-6cd7tcwgw\" (UID: \"6f6422f2-f1b6-433f-93a3-52edd828c682\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6cd7tcwgw"
Oct 30 13:46:31.262184 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:31.262021    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f6422f2-f1b6-433f-93a3-52edd828c682-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-310546-6cd7tcwgw\" (UID: \"6f6422f2-f1b6-433f-93a3-52edd828c682\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6cd7tcwgw"
Oct 30 13:46:31.262610 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:31.262589    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6f6422f2-f1b6-433f-93a3-52edd828c682-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-310546-6cd7tcwgw\" (UID: \"6f6422f2-f1b6-433f-93a3-52edd828c682\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6cd7tcwgw"
Oct 30 13:46:31.282171 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:31.282122    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrwws\" (UniqueName: \"kubernetes.io/projected/6f6422f2-f1b6-433f-93a3-52edd828c682-kube-api-access-vrwws\") pod \"payload-processing-external-model-e2e-extmod-a-310546-6cd7tcwgw\" (UID: \"6f6422f2-f1b6-433f-93a3-52edd828c682\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6cd7tcwgw"
Oct 30 13:46:31.360794 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:31.360694    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6cd7tcwgw"
Oct 30 13:46:31.675242 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:31.675208    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6cd7tcwgw"]
Oct 30 13:46:31.677676 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:46:31.677645    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f6422f2_f1b6_433f_93a3_52edd828c682.slice/crio-04b57a9789e0a0ba65578e2996c2e94b32c8d704d277766cc05cbe6e363c5f2b WatchSource:0}: Error finding container 04b57a9789e0a0ba65578e2996c2e94b32c8d704d277766cc05cbe6e363c5f2b: Status 404 returned error can't find the container with id 04b57a9789e0a0ba65578e2996c2e94b32c8d704d277766cc05cbe6e363c5f2b
Oct 30 13:46:31.901556 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:31.901518    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6558x68zp"]
Oct 30 13:46:31.907609 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:31.907579    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6558x68zp"
Oct 30 13:46:31.927990 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:31.927956    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6558x68zp"]
Oct 30 13:46:31.932590 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:31.932417    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e3eb98\"/\"payload-processing-external-model-plugins-e2e-extmod-b-e3eb98\""
Oct 30 13:46:31.932590 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:31.932447    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e3eb98\"/\"kube-root-ca.crt\""
Oct 30 13:46:31.932590 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:31.932457    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e3eb98\"/\"routing-overlay\""
Oct 30 13:46:31.932590 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:31.932480    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e3eb98\"/\"openshift-service-ca.crt\""
Oct 30 13:46:31.932590 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:31.932496    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e3eb98\"/\"agc-cred-8d03b77200d7\""
Oct 30 13:46:31.932590 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:31.932584    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e3eb98\"/\"payload-processing-external-model-e2e--2547b850-dockercfg-zlgbr\""
Oct 30 13:46:32.089153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:32.085095    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4e01c3f-d5a5-47c8-8d75-33883b7b3d74-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-6558x68zp\" (UID: \"a4e01c3f-d5a5-47c8-8d75-33883b7b3d74\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6558x68zp"
Oct 30 13:46:32.089153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:32.085222    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a4e01c3f-d5a5-47c8-8d75-33883b7b3d74-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-6558x68zp\" (UID: \"a4e01c3f-d5a5-47c8-8d75-33883b7b3d74\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6558x68zp"
Oct 30 13:46:32.089153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:32.085257    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a4e01c3f-d5a5-47c8-8d75-33883b7b3d74-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-6558x68zp\" (UID: \"a4e01c3f-d5a5-47c8-8d75-33883b7b3d74\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6558x68zp"
Oct 30 13:46:32.089153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:32.085558    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5xxx\" (UniqueName: \"kubernetes.io/projected/a4e01c3f-d5a5-47c8-8d75-33883b7b3d74-kube-api-access-k5xxx\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-6558x68zp\" (UID: \"a4e01c3f-d5a5-47c8-8d75-33883b7b3d74\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6558x68zp"
Oct 30 13:46:32.187028 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:32.186935    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5xxx\" (UniqueName: \"kubernetes.io/projected/a4e01c3f-d5a5-47c8-8d75-33883b7b3d74-kube-api-access-k5xxx\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-6558x68zp\" (UID: \"a4e01c3f-d5a5-47c8-8d75-33883b7b3d74\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6558x68zp"
Oct 30 13:46:32.187214 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:32.187026    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4e01c3f-d5a5-47c8-8d75-33883b7b3d74-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-6558x68zp\" (UID: \"a4e01c3f-d5a5-47c8-8d75-33883b7b3d74\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6558x68zp"
Oct 30 13:46:32.187214 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:32.187086    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a4e01c3f-d5a5-47c8-8d75-33883b7b3d74-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-6558x68zp\" (UID: \"a4e01c3f-d5a5-47c8-8d75-33883b7b3d74\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6558x68zp"
Oct 30 13:46:32.187214 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:32.187111    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a4e01c3f-d5a5-47c8-8d75-33883b7b3d74-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-6558x68zp\" (UID: \"a4e01c3f-d5a5-47c8-8d75-33883b7b3d74\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6558x68zp"
Oct 30 13:46:32.187860 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:32.187826    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a4e01c3f-d5a5-47c8-8d75-33883b7b3d74-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-6558x68zp\" (UID: \"a4e01c3f-d5a5-47c8-8d75-33883b7b3d74\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6558x68zp"
Oct 30 13:46:32.187954 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:32.187863    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4e01c3f-d5a5-47c8-8d75-33883b7b3d74-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-6558x68zp\" (UID: \"a4e01c3f-d5a5-47c8-8d75-33883b7b3d74\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6558x68zp"
Oct 30 13:46:32.189690 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:32.189670    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a4e01c3f-d5a5-47c8-8d75-33883b7b3d74-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-6558x68zp\" (UID: \"a4e01c3f-d5a5-47c8-8d75-33883b7b3d74\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6558x68zp"
Oct 30 13:46:32.207065 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:32.207036    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5xxx\" (UniqueName: \"kubernetes.io/projected/a4e01c3f-d5a5-47c8-8d75-33883b7b3d74-kube-api-access-k5xxx\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-6558x68zp\" (UID: \"a4e01c3f-d5a5-47c8-8d75-33883b7b3d74\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6558x68zp"
Oct 30 13:46:32.228757 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:32.228724    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6558x68zp"
Oct 30 13:46:32.543311 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:32.543275    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6558x68zp"]
Oct 30 13:46:32.543785 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:46:32.543760    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4e01c3f_d5a5_47c8_8d75_33883b7b3d74.slice/crio-6207aaaa4aeda0f6fe080b5bc11704809581dcb9f7d521c9c04184bd88d206ff WatchSource:0}: Error finding container 6207aaaa4aeda0f6fe080b5bc11704809581dcb9f7d521c9c04184bd88d206ff: Status 404 returned error can't find the container with id 6207aaaa4aeda0f6fe080b5bc11704809581dcb9f7d521c9c04184bd88d206ff
Oct 30 13:46:32.672590 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:32.672552    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6cd7tcwgw" event={"ID":"6f6422f2-f1b6-433f-93a3-52edd828c682","Type":"ContainerStarted","Data":"362173a912f5f1fcb25e3b1956c812a35c20fbbdb44a5add473ffde334ee99a0"}
Oct 30 13:46:32.672801 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:32.672623    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6cd7tcwgw" event={"ID":"6f6422f2-f1b6-433f-93a3-52edd828c682","Type":"ContainerStarted","Data":"04b57a9789e0a0ba65578e2996c2e94b32c8d704d277766cc05cbe6e363c5f2b"}
Oct 30 13:46:32.672801 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:32.672653    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6cd7tcwgw"
Oct 30 13:46:32.674739 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:32.674707    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6558x68zp" event={"ID":"a4e01c3f-d5a5-47c8-8d75-33883b7b3d74","Type":"ContainerStarted","Data":"cd50346345055674a36da0abdfbfaa80998aaf3163faa889a1e9d3958a70edef"}
Oct 30 13:46:32.674872 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:32.674761    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6558x68zp" event={"ID":"a4e01c3f-d5a5-47c8-8d75-33883b7b3d74","Type":"ContainerStarted","Data":"6207aaaa4aeda0f6fe080b5bc11704809581dcb9f7d521c9c04184bd88d206ff"}
Oct 30 13:46:32.674872 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:32.674792    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6558x68zp"
Oct 30 13:46:32.717658 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:32.717604    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6cd7tcwgw" podStartSLOduration=1.717589271 podStartE2EDuration="1.717589271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:46:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:46:32.715356516 +0000 UTC m=+2611.862623756" watchObservedRunningTime="2026-10-30 13:46:32.717589271 +0000 UTC m=+2611.864856512"
Oct 30 13:46:32.759646 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:32.759589    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6558x68zp" podStartSLOduration=1.759571928 podStartE2EDuration="1.759571928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:46:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:46:32.757107841 +0000 UTC m=+2611.904375074" watchObservedRunningTime="2026-10-30 13:46:32.759571928 +0000 UTC m=+2611.906839203"
Oct 30 13:46:43.686086 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:43.686052    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6cd7tcwgw"
Oct 30 13:46:43.686505 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:46:43.686159    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6558x68zp"
Oct 30 13:46:51.169351 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:46:51.169307    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:47:03.347878 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:03.347835    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-7864wbdnh"]
Oct 30 13:47:08.348494 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:08.348391    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-7864wbdnh" podUID="6641c45f-b1ee-4de8-98d8-dd17093cda57" containerName="payload-processing" containerID="cri-o://824e4add8d144638fc2f1aa7d5766423ca3e77639423d8251c6c3b93aad37fa3" gracePeriod=25
Oct 30 13:47:08.841672 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:08.841592    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-7864wbdnh"
Oct 30 13:47:08.994161 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:08.994097    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6641c45f-b1ee-4de8-98d8-dd17093cda57-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6641c45f-b1ee-4de8-98d8-dd17093cda57-plugins-config-volume\") pod \"6641c45f-b1ee-4de8-98d8-dd17093cda57\" (UID: \"6641c45f-b1ee-4de8-98d8-dd17093cda57\") "
Oct 30 13:47:08.994361 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:08.994259    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6641c45f-b1ee-4de8-98d8-dd17093cda57-kube-api-access-mptrk\" (UniqueName: \"kubernetes.io/projected/6641c45f-b1ee-4de8-98d8-dd17093cda57-kube-api-access-mptrk\") pod \"6641c45f-b1ee-4de8-98d8-dd17093cda57\" (UID: \"6641c45f-b1ee-4de8-98d8-dd17093cda57\") "
Oct 30 13:47:08.994446 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:08.994381    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6641c45f-b1ee-4de8-98d8-dd17093cda57-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6641c45f-b1ee-4de8-98d8-dd17093cda57-provider-credentials\") pod \"6641c45f-b1ee-4de8-98d8-dd17093cda57\" (UID: \"6641c45f-b1ee-4de8-98d8-dd17093cda57\") "
Oct 30 13:47:08.994516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:08.994444    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6641c45f-b1ee-4de8-98d8-dd17093cda57-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6641c45f-b1ee-4de8-98d8-dd17093cda57-routing-overlay-volume\") pod \"6641c45f-b1ee-4de8-98d8-dd17093cda57\" (UID: \"6641c45f-b1ee-4de8-98d8-dd17093cda57\") "
Oct 30 13:47:08.994579 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:08.994557    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6641c45f-b1ee-4de8-98d8-dd17093cda57-plugins-config-volume" pod "6641c45f-b1ee-4de8-98d8-dd17093cda57" (UID: "6641c45f-b1ee-4de8-98d8-dd17093cda57"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:47:08.995124 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:08.995096    5093 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6641c45f-b1ee-4de8-98d8-dd17093cda57-plugins-config-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:47:08.995355 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:08.995328    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6641c45f-b1ee-4de8-98d8-dd17093cda57-routing-overlay-volume" pod "6641c45f-b1ee-4de8-98d8-dd17093cda57" (UID: "6641c45f-b1ee-4de8-98d8-dd17093cda57"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:47:08.997384 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:08.997119    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6641c45f-b1ee-4de8-98d8-dd17093cda57-provider-credentials" pod "6641c45f-b1ee-4de8-98d8-dd17093cda57" (UID: "6641c45f-b1ee-4de8-98d8-dd17093cda57"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:47:08.997858 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:08.997836    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6641c45f-b1ee-4de8-98d8-dd17093cda57-kube-api-access-mptrk" pod "6641c45f-b1ee-4de8-98d8-dd17093cda57" (UID: "6641c45f-b1ee-4de8-98d8-dd17093cda57"). InnerVolumeSpecName "kube-api-access-mptrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:47:09.097495 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:09.097461    5093 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6641c45f-b1ee-4de8-98d8-dd17093cda57-routing-overlay-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:47:09.097741 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:09.097716    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mptrk\" (UniqueName: \"kubernetes.io/projected/6641c45f-b1ee-4de8-98d8-dd17093cda57-kube-api-access-mptrk\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:47:09.097741 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:09.097741    5093 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6641c45f-b1ee-4de8-98d8-dd17093cda57-provider-credentials\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:47:09.115080 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:09.115043    5093 generic.go:356] "Generic (PLEG): container finished" podID="6641c45f-b1ee-4de8-98d8-dd17093cda57" containerID="824e4add8d144638fc2f1aa7d5766423ca3e77639423d8251c6c3b93aad37fa3" exitCode=0
Oct 30 13:47:09.115271 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:09.115100    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-7864wbdnh"
Oct 30 13:47:09.115271 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:09.115085    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-7864wbdnh" event={"ID":"6641c45f-b1ee-4de8-98d8-dd17093cda57","Type":"ContainerDied","Data":"824e4add8d144638fc2f1aa7d5766423ca3e77639423d8251c6c3b93aad37fa3"}
Oct 30 13:47:09.115271 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:09.115197    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-7864wbdnh" event={"ID":"6641c45f-b1ee-4de8-98d8-dd17093cda57","Type":"ContainerDied","Data":"232296df6afefd72fe9dc2305c049a0416f3be491b6c7606816cff2d5e5916ce"}
Oct 30 13:47:09.115271 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:09.115229    5093 scope.go:122] "RemoveContainer" containerID="824e4add8d144638fc2f1aa7d5766423ca3e77639423d8251c6c3b93aad37fa3"
Oct 30 13:47:09.126664 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:09.126636    5093 scope.go:122] "RemoveContainer" containerID="824e4add8d144638fc2f1aa7d5766423ca3e77639423d8251c6c3b93aad37fa3"
Oct 30 13:47:09.126950 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:47:09.126926    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"824e4add8d144638fc2f1aa7d5766423ca3e77639423d8251c6c3b93aad37fa3\": container with ID starting with 824e4add8d144638fc2f1aa7d5766423ca3e77639423d8251c6c3b93aad37fa3 not found: ID does not exist" containerID="824e4add8d144638fc2f1aa7d5766423ca3e77639423d8251c6c3b93aad37fa3"
Oct 30 13:47:09.126998 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:09.126961    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"824e4add8d144638fc2f1aa7d5766423ca3e77639423d8251c6c3b93aad37fa3"} err="failed to get container status \"824e4add8d144638fc2f1aa7d5766423ca3e77639423d8251c6c3b93aad37fa3\": rpc error: code = NotFound desc = could not find container \"824e4add8d144638fc2f1aa7d5766423ca3e77639423d8251c6c3b93aad37fa3\": container with ID starting with 824e4add8d144638fc2f1aa7d5766423ca3e77639423d8251c6c3b93aad37fa3 not found: ID does not exist"
Oct 30 13:47:09.171333 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:09.171300    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-7864wbdnh"]
Oct 30 13:47:09.184301 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:09.184267    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-7864wbdnh"]
Oct 30 13:47:11.013584 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:11.013479    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6641c45f-b1ee-4de8-98d8-dd17093cda57" path="/var/lib/kubelet/pods/6641c45f-b1ee-4de8-98d8-dd17093cda57/volumes"
Oct 30 13:47:20.941224 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:20.941183    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5d45f69b67-7f4lb"]
Oct 30 13:47:20.942522 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:20.942488    5093 cpu_manager.go:414] "containerMap: removing container" podUID="6641c45f-b1ee-4de8-98d8-dd17093cda57" containerName="payload-processing"
Oct 30 13:47:20.942522 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:20.942517    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6641c45f-b1ee-4de8-98d8-dd17093cda57" containerName="payload-processing"
Oct 30 13:47:20.942840 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:20.942819    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="6641c45f-b1ee-4de8-98d8-dd17093cda57" containerName="payload-processing"
Oct 30 13:47:20.948203 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:20.948182    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5d45f69b67-7f4lb"
Oct 30 13:47:20.961014 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:20.960988    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d45f69b67-7f4lb"]
Oct 30 13:47:20.975457 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:20.974552    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-7hqkz\""
Oct 30 13:47:20.975457 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:20.974615    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-6a1c345c7086\""
Oct 30 13:47:20.975457 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:20.975324    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 13:47:20.975457 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:20.975397    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 13:47:20.975791 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:20.975540    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 13:47:20.977068 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:20.977043    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 13:47:21.000043 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:21.000002    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/06f7a053-694c-4a42-aef8-fcb46658b56c-routing-overlay-volume\") pod \"payload-processing-external-model-5d45f69b67-7f4lb\" (UID: \"06f7a053-694c-4a42-aef8-fcb46658b56c\") " pod="models-as-a-service/payload-processing-external-model-5d45f69b67-7f4lb"
Oct 30 13:47:21.000294 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:21.000189    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bddzs\" (UniqueName: \"kubernetes.io/projected/06f7a053-694c-4a42-aef8-fcb46658b56c-kube-api-access-bddzs\") pod \"payload-processing-external-model-5d45f69b67-7f4lb\" (UID: \"06f7a053-694c-4a42-aef8-fcb46658b56c\") " pod="models-as-a-service/payload-processing-external-model-5d45f69b67-7f4lb"
Oct 30 13:47:21.000294 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:21.000262    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06f7a053-694c-4a42-aef8-fcb46658b56c-plugins-config-volume\") pod \"payload-processing-external-model-5d45f69b67-7f4lb\" (UID: \"06f7a053-694c-4a42-aef8-fcb46658b56c\") " pod="models-as-a-service/payload-processing-external-model-5d45f69b67-7f4lb"
Oct 30 13:47:21.000294 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:21.000282    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/06f7a053-694c-4a42-aef8-fcb46658b56c-provider-credentials\") pod \"payload-processing-external-model-5d45f69b67-7f4lb\" (UID: \"06f7a053-694c-4a42-aef8-fcb46658b56c\") " pod="models-as-a-service/payload-processing-external-model-5d45f69b67-7f4lb"
Oct 30 13:47:21.116116 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:21.115794    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/06f7a053-694c-4a42-aef8-fcb46658b56c-routing-overlay-volume\") pod \"payload-processing-external-model-5d45f69b67-7f4lb\" (UID: \"06f7a053-694c-4a42-aef8-fcb46658b56c\") " pod="models-as-a-service/payload-processing-external-model-5d45f69b67-7f4lb"
Oct 30 13:47:21.116116 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:21.115900    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bddzs\" (UniqueName: \"kubernetes.io/projected/06f7a053-694c-4a42-aef8-fcb46658b56c-kube-api-access-bddzs\") pod \"payload-processing-external-model-5d45f69b67-7f4lb\" (UID: \"06f7a053-694c-4a42-aef8-fcb46658b56c\") " pod="models-as-a-service/payload-processing-external-model-5d45f69b67-7f4lb"
Oct 30 13:47:21.116116 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:21.115980    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06f7a053-694c-4a42-aef8-fcb46658b56c-plugins-config-volume\") pod \"payload-processing-external-model-5d45f69b67-7f4lb\" (UID: \"06f7a053-694c-4a42-aef8-fcb46658b56c\") " pod="models-as-a-service/payload-processing-external-model-5d45f69b67-7f4lb"
Oct 30 13:47:21.116116 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:21.116005    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/06f7a053-694c-4a42-aef8-fcb46658b56c-provider-credentials\") pod \"payload-processing-external-model-5d45f69b67-7f4lb\" (UID: \"06f7a053-694c-4a42-aef8-fcb46658b56c\") " pod="models-as-a-service/payload-processing-external-model-5d45f69b67-7f4lb"
Oct 30 13:47:21.212197 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:21.205346    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06f7a053-694c-4a42-aef8-fcb46658b56c-plugins-config-volume\") pod \"payload-processing-external-model-5d45f69b67-7f4lb\" (UID: \"06f7a053-694c-4a42-aef8-fcb46658b56c\") " pod="models-as-a-service/payload-processing-external-model-5d45f69b67-7f4lb"
Oct 30 13:47:21.213159 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:21.213074    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bddzs\" (UniqueName: \"kubernetes.io/projected/06f7a053-694c-4a42-aef8-fcb46658b56c-kube-api-access-bddzs\") pod \"payload-processing-external-model-5d45f69b67-7f4lb\" (UID: \"06f7a053-694c-4a42-aef8-fcb46658b56c\") " pod="models-as-a-service/payload-processing-external-model-5d45f69b67-7f4lb"
Oct 30 13:47:21.214164 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:21.214124    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/06f7a053-694c-4a42-aef8-fcb46658b56c-provider-credentials\") pod \"payload-processing-external-model-5d45f69b67-7f4lb\" (UID: \"06f7a053-694c-4a42-aef8-fcb46658b56c\") " pod="models-as-a-service/payload-processing-external-model-5d45f69b67-7f4lb"
Oct 30 13:47:21.215106 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:21.215080    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/06f7a053-694c-4a42-aef8-fcb46658b56c-routing-overlay-volume\") pod \"payload-processing-external-model-5d45f69b67-7f4lb\" (UID: \"06f7a053-694c-4a42-aef8-fcb46658b56c\") " pod="models-as-a-service/payload-processing-external-model-5d45f69b67-7f4lb"
Oct 30 13:47:21.255185 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:47:21.255101    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:47:21.278489 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:21.278362    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5d45f69b67-7f4lb"
Oct 30 13:47:21.591401 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:21.591363    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d45f69b67-7f4lb"]
Oct 30 13:47:21.592666 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:47:21.592636    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06f7a053_694c_4a42_aef8_fcb46658b56c.slice/crio-a1e51c249c1c79dc5ce01fda0270e90898f26e42e5ad26b3acdc0c620c387929 WatchSource:0}: Error finding container a1e51c249c1c79dc5ce01fda0270e90898f26e42e5ad26b3acdc0c620c387929: Status 404 returned error can't find the container with id a1e51c249c1c79dc5ce01fda0270e90898f26e42e5ad26b3acdc0c620c387929
Oct 30 13:47:22.255096 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:22.255050    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5d45f69b67-7f4lb" event={"ID":"06f7a053-694c-4a42-aef8-fcb46658b56c","Type":"ContainerStarted","Data":"4387439aa3e06c7a8dcf6e5f230f847aea0c58fad78cf17bc3e1eed815d48b86"}
Oct 30 13:47:22.255555 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:22.255120    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5d45f69b67-7f4lb" event={"ID":"06f7a053-694c-4a42-aef8-fcb46658b56c","Type":"ContainerStarted","Data":"a1e51c249c1c79dc5ce01fda0270e90898f26e42e5ad26b3acdc0c620c387929"}
Oct 30 13:47:22.255555 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:22.255164    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5d45f69b67-7f4lb"
Oct 30 13:47:22.299368 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:22.299308    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5d45f69b67-7f4lb" podStartSLOduration=2.299291819 podStartE2EDuration="2.299291819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:47:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:47:22.295388867 +0000 UTC m=+2661.442656108" watchObservedRunningTime="2026-10-30 13:47:22.299291819 +0000 UTC m=+2661.446559081"
Oct 30 13:47:33.269502 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:33.269468    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5d45f69b67-7f4lb"
Oct 30 13:47:49.730156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:49.729228    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9sgpb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:47:49.730156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:47:49.729312    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb" podUID="e97d49cb-0e82-4c03-8aeb-044f1c076234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:47:51.168160 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:47:51.168105    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:48:01.576742 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:01.576708    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jdj4_f93ee017-ab79-447c-90e0-5b85b71dba24/console-operator/1.log"
Oct 30 13:48:01.584073 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:01.584039    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tnm_3a675ac7-fbe1-449b-accf-bb0456ed0bd3/ovn-acl-logging/0.log"
Oct 30 13:48:01.586503 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:01.586479    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jdj4_f93ee017-ab79-447c-90e0-5b85b71dba24/console-operator/1.log"
Oct 30 13:48:01.592402 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:01.592379    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tnm_3a675ac7-fbe1-449b-accf-bb0456ed0bd3/ovn-acl-logging/0.log"
Oct 30 13:48:01.768636 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:01.768596    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d45f69b67-7f4lb"]
Oct 30 13:48:06.769386 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:06.769334    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5d45f69b67-7f4lb" podUID="06f7a053-694c-4a42-aef8-fcb46658b56c" containerName="payload-processing" containerID="cri-o://4387439aa3e06c7a8dcf6e5f230f847aea0c58fad78cf17bc3e1eed815d48b86" gracePeriod=25
Oct 30 13:48:07.240612 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:07.240578    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5d45f69b67-7f4lb"
Oct 30 13:48:07.351199 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:07.351096    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06f7a053-694c-4a42-aef8-fcb46658b56c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06f7a053-694c-4a42-aef8-fcb46658b56c-plugins-config-volume\") pod \"06f7a053-694c-4a42-aef8-fcb46658b56c\" (UID: \"06f7a053-694c-4a42-aef8-fcb46658b56c\") "
Oct 30 13:48:07.351199 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:07.351174    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06f7a053-694c-4a42-aef8-fcb46658b56c-kube-api-access-bddzs\" (UniqueName: \"kubernetes.io/projected/06f7a053-694c-4a42-aef8-fcb46658b56c-kube-api-access-bddzs\") pod \"06f7a053-694c-4a42-aef8-fcb46658b56c\" (UID: \"06f7a053-694c-4a42-aef8-fcb46658b56c\") "
Oct 30 13:48:07.351445 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:07.351424    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06f7a053-694c-4a42-aef8-fcb46658b56c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/06f7a053-694c-4a42-aef8-fcb46658b56c-provider-credentials\") pod \"06f7a053-694c-4a42-aef8-fcb46658b56c\" (UID: \"06f7a053-694c-4a42-aef8-fcb46658b56c\") "
Oct 30 13:48:07.351558 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:07.351535    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06f7a053-694c-4a42-aef8-fcb46658b56c-plugins-config-volume" pod "06f7a053-694c-4a42-aef8-fcb46658b56c" (UID: "06f7a053-694c-4a42-aef8-fcb46658b56c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:48:07.351619 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:07.351551    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06f7a053-694c-4a42-aef8-fcb46658b56c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/06f7a053-694c-4a42-aef8-fcb46658b56c-routing-overlay-volume\") pod \"06f7a053-694c-4a42-aef8-fcb46658b56c\" (UID: \"06f7a053-694c-4a42-aef8-fcb46658b56c\") "
Oct 30 13:48:07.352018 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:07.351995    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06f7a053-694c-4a42-aef8-fcb46658b56c-routing-overlay-volume" pod "06f7a053-694c-4a42-aef8-fcb46658b56c" (UID: "06f7a053-694c-4a42-aef8-fcb46658b56c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:48:07.352220 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:07.352195    5093 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/06f7a053-694c-4a42-aef8-fcb46658b56c-routing-overlay-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:48:07.352314 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:07.352216    5093 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06f7a053-694c-4a42-aef8-fcb46658b56c-plugins-config-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:48:07.354512 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:07.354480    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06f7a053-694c-4a42-aef8-fcb46658b56c-kube-api-access-bddzs" pod "06f7a053-694c-4a42-aef8-fcb46658b56c" (UID: "06f7a053-694c-4a42-aef8-fcb46658b56c"). InnerVolumeSpecName "kube-api-access-bddzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:48:07.354649 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:07.354527    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06f7a053-694c-4a42-aef8-fcb46658b56c-provider-credentials" pod "06f7a053-694c-4a42-aef8-fcb46658b56c" (UID: "06f7a053-694c-4a42-aef8-fcb46658b56c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:48:07.454055 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:07.454015    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bddzs\" (UniqueName: \"kubernetes.io/projected/06f7a053-694c-4a42-aef8-fcb46658b56c-kube-api-access-bddzs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:48:07.454055 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:07.454049    5093 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/06f7a053-694c-4a42-aef8-fcb46658b56c-provider-credentials\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:48:07.780965 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:07.780927    5093 generic.go:356] "Generic (PLEG): container finished" podID="06f7a053-694c-4a42-aef8-fcb46658b56c" containerID="4387439aa3e06c7a8dcf6e5f230f847aea0c58fad78cf17bc3e1eed815d48b86" exitCode=0
Oct 30 13:48:07.781433 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:07.780983    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5d45f69b67-7f4lb"
Oct 30 13:48:07.781433 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:07.781008    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5d45f69b67-7f4lb" event={"ID":"06f7a053-694c-4a42-aef8-fcb46658b56c","Type":"ContainerDied","Data":"4387439aa3e06c7a8dcf6e5f230f847aea0c58fad78cf17bc3e1eed815d48b86"}
Oct 30 13:48:07.781433 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:07.781070    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5d45f69b67-7f4lb" event={"ID":"06f7a053-694c-4a42-aef8-fcb46658b56c","Type":"ContainerDied","Data":"a1e51c249c1c79dc5ce01fda0270e90898f26e42e5ad26b3acdc0c620c387929"}
Oct 30 13:48:07.781433 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:07.781098    5093 scope.go:122] "RemoveContainer" containerID="4387439aa3e06c7a8dcf6e5f230f847aea0c58fad78cf17bc3e1eed815d48b86"
Oct 30 13:48:07.798529 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:07.798501    5093 scope.go:122] "RemoveContainer" containerID="4387439aa3e06c7a8dcf6e5f230f847aea0c58fad78cf17bc3e1eed815d48b86"
Oct 30 13:48:07.798847 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:48:07.798820    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4387439aa3e06c7a8dcf6e5f230f847aea0c58fad78cf17bc3e1eed815d48b86\": container with ID starting with 4387439aa3e06c7a8dcf6e5f230f847aea0c58fad78cf17bc3e1eed815d48b86 not found: ID does not exist" containerID="4387439aa3e06c7a8dcf6e5f230f847aea0c58fad78cf17bc3e1eed815d48b86"
Oct 30 13:48:07.798916 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:07.798859    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4387439aa3e06c7a8dcf6e5f230f847aea0c58fad78cf17bc3e1eed815d48b86"} err="failed to get container status \"4387439aa3e06c7a8dcf6e5f230f847aea0c58fad78cf17bc3e1eed815d48b86\": rpc error: code = NotFound desc = could not find container \"4387439aa3e06c7a8dcf6e5f230f847aea0c58fad78cf17bc3e1eed815d48b86\": container with ID starting with 4387439aa3e06c7a8dcf6e5f230f847aea0c58fad78cf17bc3e1eed815d48b86 not found: ID does not exist"
Oct 30 13:48:07.887163 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:07.885544    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d45f69b67-7f4lb"]
Oct 30 13:48:07.890482 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:07.890449    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d45f69b67-7f4lb"]
Oct 30 13:48:08.034399 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:08.034316    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-6f4fph8w2"]
Oct 30 13:48:08.035177 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:08.035158    5093 cpu_manager.go:414] "containerMap: removing container" podUID="06f7a053-694c-4a42-aef8-fcb46658b56c" containerName="payload-processing"
Oct 30 13:48:08.035235 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:08.035180    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06f7a053-694c-4a42-aef8-fcb46658b56c" containerName="payload-processing"
Oct 30 13:48:08.035324 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:08.035313    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="06f7a053-694c-4a42-aef8-fcb46658b56c" containerName="payload-processing"
Oct 30 13:48:08.040491 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:08.040467    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-6f4fph8w2"
Oct 30 13:48:08.064354 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:08.064325    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-844856\"/\"agc-cred-9f935abd3088\""
Oct 30 13:48:08.064560 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:08.064377    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-844856\"/\"payload-processing-external-model-e2e--cbdef895-dockercfg-n5sv4\""
Oct 30 13:48:08.070567 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:08.070539    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-6f4fph8w2"]
Oct 30 13:48:08.094213 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:08.094104    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-844856\"/\"kube-root-ca.crt\""
Oct 30 13:48:08.094381 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:08.094236    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-844856\"/\"routing-overlay\""
Oct 30 13:48:08.094381 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:08.094284    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-844856\"/\"payload-processing-external-model-plugins-e2e-extmod-b-844856\""
Oct 30 13:48:08.094381 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:08.094369    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-844856\"/\"openshift-service-ca.crt\""
Oct 30 13:48:08.188187 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:08.175972    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e23ff56-28e7-41ea-8a54-a0ee68f236cf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-844856-6f4fph8w2\" (UID: \"0e23ff56-28e7-41ea-8a54-a0ee68f236cf\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-6f4fph8w2"
Oct 30 13:48:08.188187 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:08.184466    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0e23ff56-28e7-41ea-8a54-a0ee68f236cf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-844856-6f4fph8w2\" (UID: \"0e23ff56-28e7-41ea-8a54-a0ee68f236cf\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-6f4fph8w2"
Oct 30 13:48:08.188187 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:08.184596    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0e23ff56-28e7-41ea-8a54-a0ee68f236cf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-844856-6f4fph8w2\" (UID: \"0e23ff56-28e7-41ea-8a54-a0ee68f236cf\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-6f4fph8w2"
Oct 30 13:48:08.188187 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:08.184649    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmpkw\" (UniqueName: \"kubernetes.io/projected/0e23ff56-28e7-41ea-8a54-a0ee68f236cf-kube-api-access-mmpkw\") pod \"payload-processing-external-model-e2e-extmod-b-844856-6f4fph8w2\" (UID: \"0e23ff56-28e7-41ea-8a54-a0ee68f236cf\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-6f4fph8w2"
Oct 30 13:48:08.286122 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:08.286013    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e23ff56-28e7-41ea-8a54-a0ee68f236cf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-844856-6f4fph8w2\" (UID: \"0e23ff56-28e7-41ea-8a54-a0ee68f236cf\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-6f4fph8w2"
Oct 30 13:48:08.286333 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:08.286166    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0e23ff56-28e7-41ea-8a54-a0ee68f236cf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-844856-6f4fph8w2\" (UID: \"0e23ff56-28e7-41ea-8a54-a0ee68f236cf\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-6f4fph8w2"
Oct 30 13:48:08.286333 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:08.286266    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0e23ff56-28e7-41ea-8a54-a0ee68f236cf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-844856-6f4fph8w2\" (UID: \"0e23ff56-28e7-41ea-8a54-a0ee68f236cf\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-6f4fph8w2"
Oct 30 13:48:08.286333 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:08.286319    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmpkw\" (UniqueName: \"kubernetes.io/projected/0e23ff56-28e7-41ea-8a54-a0ee68f236cf-kube-api-access-mmpkw\") pod \"payload-processing-external-model-e2e-extmod-b-844856-6f4fph8w2\" (UID: \"0e23ff56-28e7-41ea-8a54-a0ee68f236cf\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-6f4fph8w2"
Oct 30 13:48:08.287058 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:08.287021    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e23ff56-28e7-41ea-8a54-a0ee68f236cf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-844856-6f4fph8w2\" (UID: \"0e23ff56-28e7-41ea-8a54-a0ee68f236cf\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-6f4fph8w2"
Oct 30 13:48:08.287370 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:08.287344    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0e23ff56-28e7-41ea-8a54-a0ee68f236cf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-844856-6f4fph8w2\" (UID: \"0e23ff56-28e7-41ea-8a54-a0ee68f236cf\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-6f4fph8w2"
Oct 30 13:48:08.289523 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:08.289491    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0e23ff56-28e7-41ea-8a54-a0ee68f236cf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-844856-6f4fph8w2\" (UID: \"0e23ff56-28e7-41ea-8a54-a0ee68f236cf\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-6f4fph8w2"
Oct 30 13:48:08.316885 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:08.316856    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmpkw\" (UniqueName: \"kubernetes.io/projected/0e23ff56-28e7-41ea-8a54-a0ee68f236cf-kube-api-access-mmpkw\") pod \"payload-processing-external-model-e2e-extmod-b-844856-6f4fph8w2\" (UID: \"0e23ff56-28e7-41ea-8a54-a0ee68f236cf\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-6f4fph8w2"
Oct 30 13:48:08.355911 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:08.355876    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-6f4fph8w2"
Oct 30 13:48:08.693698 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:08.693481    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-6f4fph8w2"]
Oct 30 13:48:08.693863 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:48:08.693776    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e23ff56_28e7_41ea_8a54_a0ee68f236cf.slice/crio-c212c35b3a8138594310c9f463de7bb7bbb0939dd69b9d519e92e6bf38b0d19b WatchSource:0}: Error finding container c212c35b3a8138594310c9f463de7bb7bbb0939dd69b9d519e92e6bf38b0d19b: Status 404 returned error can't find the container with id c212c35b3a8138594310c9f463de7bb7bbb0939dd69b9d519e92e6bf38b0d19b
Oct 30 13:48:08.794892 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:08.794841    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-6f4fph8w2" event={"ID":"0e23ff56-28e7-41ea-8a54-a0ee68f236cf","Type":"ContainerStarted","Data":"719fb760d7335678e18733058a65eeb36f0c888b66c6452e6c4275dd89cba1d2"}
Oct 30 13:48:08.795340 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:08.794914    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-6f4fph8w2" event={"ID":"0e23ff56-28e7-41ea-8a54-a0ee68f236cf","Type":"ContainerStarted","Data":"c212c35b3a8138594310c9f463de7bb7bbb0939dd69b9d519e92e6bf38b0d19b"}
Oct 30 13:48:08.795340 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:08.794945    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-6f4fph8w2"
Oct 30 13:48:08.855439 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:08.855373    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-6f4fph8w2" podStartSLOduration=1.855354237 podStartE2EDuration="1.855354237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:48:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:08.849395691 +0000 UTC m=+2707.996662933" watchObservedRunningTime="2026-10-30 13:48:08.855354237 +0000 UTC m=+2708.002621479"
Oct 30 13:48:09.041355 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:09.038677    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06f7a053-694c-4a42-aef8-fcb46658b56c" path="/var/lib/kubelet/pods/06f7a053-694c-4a42-aef8-fcb46658b56c/volumes"
Oct 30 13:48:15.729514 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:15.729476    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-6f4fph8w2"]
Oct 30 13:48:15.746555 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:15.746526    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-6f4fph8w2"
Oct 30 13:48:20.730715 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:20.730658    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-6f4fph8w2" podUID="0e23ff56-28e7-41ea-8a54-a0ee68f236cf" containerName="payload-processing" containerID="cri-o://719fb760d7335678e18733058a65eeb36f0c888b66c6452e6c4275dd89cba1d2" gracePeriod=25
Oct 30 13:48:20.906234 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:20.906188    5093 generic.go:356] "Generic (PLEG): container finished" podID="0e23ff56-28e7-41ea-8a54-a0ee68f236cf" containerID="719fb760d7335678e18733058a65eeb36f0c888b66c6452e6c4275dd89cba1d2" exitCode=0
Oct 30 13:48:20.906400 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:20.906239    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-6f4fph8w2" event={"ID":"0e23ff56-28e7-41ea-8a54-a0ee68f236cf","Type":"ContainerDied","Data":"719fb760d7335678e18733058a65eeb36f0c888b66c6452e6c4275dd89cba1d2"}
Oct 30 13:48:21.052330 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:21.052298    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-6f4fph8w2"
Oct 30 13:48:21.137460 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:21.137407    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e23ff56-28e7-41ea-8a54-a0ee68f236cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e23ff56-28e7-41ea-8a54-a0ee68f236cf-plugins-config-volume\") pod \"0e23ff56-28e7-41ea-8a54-a0ee68f236cf\" (UID: \"0e23ff56-28e7-41ea-8a54-a0ee68f236cf\") "
Oct 30 13:48:21.138429 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:21.138382    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e23ff56-28e7-41ea-8a54-a0ee68f236cf-kube-api-access-mmpkw\" (UniqueName: \"kubernetes.io/projected/0e23ff56-28e7-41ea-8a54-a0ee68f236cf-kube-api-access-mmpkw\") pod \"0e23ff56-28e7-41ea-8a54-a0ee68f236cf\" (UID: \"0e23ff56-28e7-41ea-8a54-a0ee68f236cf\") "
Oct 30 13:48:21.138969 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:21.138950    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e23ff56-28e7-41ea-8a54-a0ee68f236cf-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0e23ff56-28e7-41ea-8a54-a0ee68f236cf-routing-overlay-volume\") pod \"0e23ff56-28e7-41ea-8a54-a0ee68f236cf\" (UID: \"0e23ff56-28e7-41ea-8a54-a0ee68f236cf\") "
Oct 30 13:48:21.140231 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:21.138173    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e23ff56-28e7-41ea-8a54-a0ee68f236cf-plugins-config-volume" pod "0e23ff56-28e7-41ea-8a54-a0ee68f236cf" (UID: "0e23ff56-28e7-41ea-8a54-a0ee68f236cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:48:21.140390 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:21.140012    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e23ff56-28e7-41ea-8a54-a0ee68f236cf-routing-overlay-volume" pod "0e23ff56-28e7-41ea-8a54-a0ee68f236cf" (UID: "0e23ff56-28e7-41ea-8a54-a0ee68f236cf"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:48:21.141038 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:21.140619    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e23ff56-28e7-41ea-8a54-a0ee68f236cf-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0e23ff56-28e7-41ea-8a54-a0ee68f236cf-provider-credentials\") pod \"0e23ff56-28e7-41ea-8a54-a0ee68f236cf\" (UID: \"0e23ff56-28e7-41ea-8a54-a0ee68f236cf\") "
Oct 30 13:48:21.144688 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:21.144668    5093 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e23ff56-28e7-41ea-8a54-a0ee68f236cf-plugins-config-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:48:21.144811 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:21.144798    5093 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0e23ff56-28e7-41ea-8a54-a0ee68f236cf-routing-overlay-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:48:21.147572 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:21.147541    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e23ff56-28e7-41ea-8a54-a0ee68f236cf-provider-credentials" pod "0e23ff56-28e7-41ea-8a54-a0ee68f236cf" (UID: "0e23ff56-28e7-41ea-8a54-a0ee68f236cf"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:48:21.147814 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:21.147773    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e23ff56-28e7-41ea-8a54-a0ee68f236cf-kube-api-access-mmpkw" pod "0e23ff56-28e7-41ea-8a54-a0ee68f236cf" (UID: "0e23ff56-28e7-41ea-8a54-a0ee68f236cf"). InnerVolumeSpecName "kube-api-access-mmpkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:48:21.176869 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:48:21.176825    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:48:21.246113 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:21.246075    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmpkw\" (UniqueName: \"kubernetes.io/projected/0e23ff56-28e7-41ea-8a54-a0ee68f236cf-kube-api-access-mmpkw\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:48:21.246113 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:21.246110    5093 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0e23ff56-28e7-41ea-8a54-a0ee68f236cf-provider-credentials\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:48:21.916279 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:21.916244    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-6f4fph8w2" event={"ID":"0e23ff56-28e7-41ea-8a54-a0ee68f236cf","Type":"ContainerDied","Data":"c212c35b3a8138594310c9f463de7bb7bbb0939dd69b9d519e92e6bf38b0d19b"}
Oct 30 13:48:21.916279 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:21.916280    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-6f4fph8w2"
Oct 30 13:48:21.916780 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:21.916312    5093 scope.go:122] "RemoveContainer" containerID="719fb760d7335678e18733058a65eeb36f0c888b66c6452e6c4275dd89cba1d2"
Oct 30 13:48:21.977055 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:21.977017    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-6f4fph8w2"]
Oct 30 13:48:21.981222 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:21.981198    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-6f4fph8w2"]
Oct 30 13:48:23.006027 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:23.005982    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e23ff56-28e7-41ea-8a54-a0ee68f236cf" path="/var/lib/kubelet/pods/0e23ff56-28e7-41ea-8a54-a0ee68f236cf/volumes"
Oct 30 13:48:24.035977 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:24.035935    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx"]
Oct 30 13:48:24.066153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:24.044810    5093 cpu_manager.go:414] "containerMap: removing container" podUID="0e23ff56-28e7-41ea-8a54-a0ee68f236cf" containerName="payload-processing"
Oct 30 13:48:24.072121 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:24.066384    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e23ff56-28e7-41ea-8a54-a0ee68f236cf" containerName="payload-processing"
Oct 30 13:48:24.072846 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:24.072804    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e23ff56-28e7-41ea-8a54-a0ee68f236cf" containerName="payload-processing"
Oct 30 13:48:24.079052 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:24.079009    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx"
Oct 30 13:48:24.081633 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:24.081604    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx"]
Oct 30 13:48:24.096039 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:24.096004    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-5494c9-openshift-default-dockercfg-96snz\""
Oct 30 13:48:24.125102 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:24.125066    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d474316d-631a-4b79-baaf-2e19fb16a211-workload-socket\") pod \"e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx\" (UID: \"d474316d-631a-4b79-baaf-2e19fb16a211\") " pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx"
Oct 30 13:48:24.125855 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:24.125207    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d474316d-631a-4b79-baaf-2e19fb16a211-workload-certs\") pod \"e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx\" (UID: \"d474316d-631a-4b79-baaf-2e19fb16a211\") " pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx"
Oct 30 13:48:24.125855 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:24.125294    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d474316d-631a-4b79-baaf-2e19fb16a211-istio-podinfo\") pod \"e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx\" (UID: \"d474316d-631a-4b79-baaf-2e19fb16a211\") " pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx"
Oct 30 13:48:24.125855 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:24.125323    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d474316d-631a-4b79-baaf-2e19fb16a211-istio-token\") pod \"e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx\" (UID: \"d474316d-631a-4b79-baaf-2e19fb16a211\") " pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx"
Oct 30 13:48:24.125855 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:24.125399    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skms5\" (UniqueName: \"kubernetes.io/projected/d474316d-631a-4b79-baaf-2e19fb16a211-kube-api-access-skms5\") pod \"e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx\" (UID: \"d474316d-631a-4b79-baaf-2e19fb16a211\") " pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx"
Oct 30 13:48:24.125855 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:24.125441    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d474316d-631a-4b79-baaf-2e19fb16a211-istio-data\") pod \"e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx\" (UID: \"d474316d-631a-4b79-baaf-2e19fb16a211\") " pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx"
Oct 30 13:48:24.125855 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:24.125595    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d474316d-631a-4b79-baaf-2e19fb16a211-credential-socket\") pod \"e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx\" (UID: \"d474316d-631a-4b79-baaf-2e19fb16a211\") " pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx"
Oct 30 13:48:24.125855 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:24.125625    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d474316d-631a-4b79-baaf-2e19fb16a211-istio-envoy\") pod \"e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx\" (UID: \"d474316d-631a-4b79-baaf-2e19fb16a211\") " pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx"
Oct 30 13:48:24.125855 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:24.125667    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d474316d-631a-4b79-baaf-2e19fb16a211-istiod-ca-cert\") pod \"e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx\" (UID: \"d474316d-631a-4b79-baaf-2e19fb16a211\") " pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx"
Oct 30 13:48:24.258153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:24.229848    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skms5\" (UniqueName: \"kubernetes.io/projected/d474316d-631a-4b79-baaf-2e19fb16a211-kube-api-access-skms5\") pod \"e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx\" (UID: \"d474316d-631a-4b79-baaf-2e19fb16a211\") " pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx"
Oct 30 13:48:24.258153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:24.255318    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d474316d-631a-4b79-baaf-2e19fb16a211-istio-data\") pod \"e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx\" (UID: \"d474316d-631a-4b79-baaf-2e19fb16a211\") " pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx"
Oct 30 13:48:24.258153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:24.255537    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d474316d-631a-4b79-baaf-2e19fb16a211-credential-socket\") pod \"e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx\" (UID: \"d474316d-631a-4b79-baaf-2e19fb16a211\") " pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx"
Oct 30 13:48:24.258153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:24.255578    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d474316d-631a-4b79-baaf-2e19fb16a211-istio-envoy\") pod \"e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx\" (UID: \"d474316d-631a-4b79-baaf-2e19fb16a211\") " pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx"
Oct 30 13:48:24.258153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:24.255623    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d474316d-631a-4b79-baaf-2e19fb16a211-istiod-ca-cert\") pod \"e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx\" (UID: \"d474316d-631a-4b79-baaf-2e19fb16a211\") " pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx"
Oct 30 13:48:24.258153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:24.255766    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d474316d-631a-4b79-baaf-2e19fb16a211-workload-socket\") pod \"e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx\" (UID: \"d474316d-631a-4b79-baaf-2e19fb16a211\") " pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx"
Oct 30 13:48:24.258153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:24.255813    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d474316d-631a-4b79-baaf-2e19fb16a211-workload-certs\") pod \"e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx\" (UID: \"d474316d-631a-4b79-baaf-2e19fb16a211\") " pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx"
Oct 30 13:48:24.258153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:24.255881    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d474316d-631a-4b79-baaf-2e19fb16a211-istio-podinfo\") pod \"e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx\" (UID: \"d474316d-631a-4b79-baaf-2e19fb16a211\") " pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx"
Oct 30 13:48:24.258153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:24.255910    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d474316d-631a-4b79-baaf-2e19fb16a211-istio-token\") pod \"e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx\" (UID: \"d474316d-631a-4b79-baaf-2e19fb16a211\") " pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx"
Oct 30 13:48:24.258153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:24.256757    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d474316d-631a-4b79-baaf-2e19fb16a211-istio-data\") pod \"e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx\" (UID: \"d474316d-631a-4b79-baaf-2e19fb16a211\") " pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx"
Oct 30 13:48:24.258153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:24.257226    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d474316d-631a-4b79-baaf-2e19fb16a211-credential-socket\") pod \"e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx\" (UID: \"d474316d-631a-4b79-baaf-2e19fb16a211\") " pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx"
Oct 30 13:48:24.280566 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:24.268847    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d474316d-631a-4b79-baaf-2e19fb16a211-workload-socket\") pod \"e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx\" (UID: \"d474316d-631a-4b79-baaf-2e19fb16a211\") " pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx"
Oct 30 13:48:24.280566 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:24.269533    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d474316d-631a-4b79-baaf-2e19fb16a211-istiod-ca-cert\") pod \"e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx\" (UID: \"d474316d-631a-4b79-baaf-2e19fb16a211\") " pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx"
Oct 30 13:48:24.280566 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:24.269822    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d474316d-631a-4b79-baaf-2e19fb16a211-workload-certs\") pod \"e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx\" (UID: \"d474316d-631a-4b79-baaf-2e19fb16a211\") " pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx"
Oct 30 13:48:24.287231 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:24.284840    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d474316d-631a-4b79-baaf-2e19fb16a211-istio-envoy\") pod \"e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx\" (UID: \"d474316d-631a-4b79-baaf-2e19fb16a211\") " pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx"
Oct 30 13:48:24.324148 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:24.320883    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d474316d-631a-4b79-baaf-2e19fb16a211-istio-podinfo\") pod \"e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx\" (UID: \"d474316d-631a-4b79-baaf-2e19fb16a211\") " pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx"
Oct 30 13:48:24.332151 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:24.328897    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skms5\" (UniqueName: \"kubernetes.io/projected/d474316d-631a-4b79-baaf-2e19fb16a211-kube-api-access-skms5\") pod \"e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx\" (UID: \"d474316d-631a-4b79-baaf-2e19fb16a211\") " pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx"
Oct 30 13:48:24.332151 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:24.329930    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d474316d-631a-4b79-baaf-2e19fb16a211-istio-token\") pod \"e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx\" (UID: \"d474316d-631a-4b79-baaf-2e19fb16a211\") " pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx"
Oct 30 13:48:24.416836 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:24.416801    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx"
Oct 30 13:48:24.740929 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:24.740894    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx"]
Oct 30 13:48:24.741799 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:48:24.741768    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd474316d_631a_4b79_baaf_2e19fb16a211.slice/crio-744acaf468401f798855d3d315d0ec42685bd784c8274a1fbf4dbdd316112a8e WatchSource:0}: Error finding container 744acaf468401f798855d3d315d0ec42685bd784c8274a1fbf4dbdd316112a8e: Status 404 returned error can't find the container with id 744acaf468401f798855d3d315d0ec42685bd784c8274a1fbf4dbdd316112a8e
Oct 30 13:48:24.744891 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:24.744852    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:48:24.744995 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:24.744927    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:48:24.744995 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:24.744960    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:48:24.951404 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:24.950428    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx" event={"ID":"d474316d-631a-4b79-baaf-2e19fb16a211","Type":"ContainerStarted","Data":"ef9cc275b940404c3b166458727f4ead429a04c9b6dc15604b44792bf5a197ad"}
Oct 30 13:48:24.951404 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:24.950502    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx" event={"ID":"d474316d-631a-4b79-baaf-2e19fb16a211","Type":"ContainerStarted","Data":"744acaf468401f798855d3d315d0ec42685bd784c8274a1fbf4dbdd316112a8e"}
Oct 30 13:48:25.001726 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:25.001607    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx" podStartSLOduration=2.001586461 podStartE2EDuration="2.001586461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:48:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:24.999651895 +0000 UTC m=+2724.146919137" watchObservedRunningTime="2026-10-30 13:48:25.001586461 +0000 UTC m=+2724.148853716"
Oct 30 13:48:25.417812 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:25.417715    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx"
Oct 30 13:48:25.419627 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:25.419596    5093 patch_prober.go:29] interesting pod/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 30 13:48:25.419767 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:25.419660    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx" podUID="d474316d-631a-4b79-baaf-2e19fb16a211" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 30 13:48:26.436690 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:26.436647    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx"
Oct 30 13:48:26.977278 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:26.977242    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx"
Oct 30 13:48:26.979685 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:26.979646    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx"
Oct 30 13:48:40.779342 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:40.779253    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v"]
Oct 30 13:48:40.784275 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:40.784247    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v"
Oct 30 13:48:40.806466 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:40.804443    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-51605e-openshift-default-dockercfg-9s7v6\""
Oct 30 13:48:40.824191 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:40.823949    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v"]
Oct 30 13:48:40.825710 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:40.825676    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4248f1a1-17bb-439c-80e5-8952de6299eb-istio-data\") pod \"e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v\" (UID: \"4248f1a1-17bb-439c-80e5-8952de6299eb\") " pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v"
Oct 30 13:48:40.825869 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:40.825738    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4248f1a1-17bb-439c-80e5-8952de6299eb-istiod-ca-cert\") pod \"e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v\" (UID: \"4248f1a1-17bb-439c-80e5-8952de6299eb\") " pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v"
Oct 30 13:48:40.825869 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:40.825812    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4248f1a1-17bb-439c-80e5-8952de6299eb-istio-envoy\") pod \"e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v\" (UID: \"4248f1a1-17bb-439c-80e5-8952de6299eb\") " pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v"
Oct 30 13:48:40.825973 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:40.825932    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4248f1a1-17bb-439c-80e5-8952de6299eb-workload-certs\") pod \"e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v\" (UID: \"4248f1a1-17bb-439c-80e5-8952de6299eb\") " pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v"
Oct 30 13:48:40.825973 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:40.825960    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4248f1a1-17bb-439c-80e5-8952de6299eb-istio-token\") pod \"e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v\" (UID: \"4248f1a1-17bb-439c-80e5-8952de6299eb\") " pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v"
Oct 30 13:48:40.826083 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:40.826004    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4248f1a1-17bb-439c-80e5-8952de6299eb-workload-socket\") pod \"e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v\" (UID: \"4248f1a1-17bb-439c-80e5-8952de6299eb\") " pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v"
Oct 30 13:48:40.826083 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:40.826042    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8xxl\" (UniqueName: \"kubernetes.io/projected/4248f1a1-17bb-439c-80e5-8952de6299eb-kube-api-access-g8xxl\") pod \"e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v\" (UID: \"4248f1a1-17bb-439c-80e5-8952de6299eb\") " pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v"
Oct 30 13:48:40.826083 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:40.826069    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4248f1a1-17bb-439c-80e5-8952de6299eb-credential-socket\") pod \"e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v\" (UID: \"4248f1a1-17bb-439c-80e5-8952de6299eb\") " pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v"
Oct 30 13:48:40.826296 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:40.826101    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4248f1a1-17bb-439c-80e5-8952de6299eb-istio-podinfo\") pod \"e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v\" (UID: \"4248f1a1-17bb-439c-80e5-8952de6299eb\") " pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v"
Oct 30 13:48:40.929936 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:40.927456    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4248f1a1-17bb-439c-80e5-8952de6299eb-workload-certs\") pod \"e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v\" (UID: \"4248f1a1-17bb-439c-80e5-8952de6299eb\") " pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v"
Oct 30 13:48:40.929936 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:40.927512    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4248f1a1-17bb-439c-80e5-8952de6299eb-istio-token\") pod \"e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v\" (UID: \"4248f1a1-17bb-439c-80e5-8952de6299eb\") " pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v"
Oct 30 13:48:40.929936 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:40.927556    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4248f1a1-17bb-439c-80e5-8952de6299eb-workload-socket\") pod \"e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v\" (UID: \"4248f1a1-17bb-439c-80e5-8952de6299eb\") " pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v"
Oct 30 13:48:40.929936 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:40.927600    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8xxl\" (UniqueName: \"kubernetes.io/projected/4248f1a1-17bb-439c-80e5-8952de6299eb-kube-api-access-g8xxl\") pod \"e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v\" (UID: \"4248f1a1-17bb-439c-80e5-8952de6299eb\") " pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v"
Oct 30 13:48:40.929936 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:40.927629    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4248f1a1-17bb-439c-80e5-8952de6299eb-credential-socket\") pod \"e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v\" (UID: \"4248f1a1-17bb-439c-80e5-8952de6299eb\") " pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v"
Oct 30 13:48:40.929936 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:40.927669    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4248f1a1-17bb-439c-80e5-8952de6299eb-istio-podinfo\") pod \"e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v\" (UID: \"4248f1a1-17bb-439c-80e5-8952de6299eb\") " pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v"
Oct 30 13:48:40.929936 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:40.927794    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4248f1a1-17bb-439c-80e5-8952de6299eb-istio-data\") pod \"e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v\" (UID: \"4248f1a1-17bb-439c-80e5-8952de6299eb\") " pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v"
Oct 30 13:48:40.929936 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:40.927841    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4248f1a1-17bb-439c-80e5-8952de6299eb-istiod-ca-cert\") pod \"e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v\" (UID: \"4248f1a1-17bb-439c-80e5-8952de6299eb\") " pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v"
Oct 30 13:48:40.929936 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:40.927945    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4248f1a1-17bb-439c-80e5-8952de6299eb-istio-envoy\") pod \"e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v\" (UID: \"4248f1a1-17bb-439c-80e5-8952de6299eb\") " pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v"
Oct 30 13:48:40.970254 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:40.947027    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4248f1a1-17bb-439c-80e5-8952de6299eb-workload-certs\") pod \"e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v\" (UID: \"4248f1a1-17bb-439c-80e5-8952de6299eb\") " pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v"
Oct 30 13:48:40.970254 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:40.947579    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4248f1a1-17bb-439c-80e5-8952de6299eb-workload-socket\") pod \"e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v\" (UID: \"4248f1a1-17bb-439c-80e5-8952de6299eb\") " pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v"
Oct 30 13:48:40.970254 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:40.955857    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4248f1a1-17bb-439c-80e5-8952de6299eb-istiod-ca-cert\") pod \"e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v\" (UID: \"4248f1a1-17bb-439c-80e5-8952de6299eb\") " pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v"
Oct 30 13:48:40.970254 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:40.956110    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4248f1a1-17bb-439c-80e5-8952de6299eb-credential-socket\") pod \"e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v\" (UID: \"4248f1a1-17bb-439c-80e5-8952de6299eb\") " pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v"
Oct 30 13:48:40.972775 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:40.972745    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4248f1a1-17bb-439c-80e5-8952de6299eb-istio-envoy\") pod \"e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v\" (UID: \"4248f1a1-17bb-439c-80e5-8952de6299eb\") " pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v"
Oct 30 13:48:40.981064 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:40.980999    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8xxl\" (UniqueName: \"kubernetes.io/projected/4248f1a1-17bb-439c-80e5-8952de6299eb-kube-api-access-g8xxl\") pod \"e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v\" (UID: \"4248f1a1-17bb-439c-80e5-8952de6299eb\") " pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v"
Oct 30 13:48:40.988214 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:40.987562    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4248f1a1-17bb-439c-80e5-8952de6299eb-istio-data\") pod \"e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v\" (UID: \"4248f1a1-17bb-439c-80e5-8952de6299eb\") " pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v"
Oct 30 13:48:40.997549 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:40.989851    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4248f1a1-17bb-439c-80e5-8952de6299eb-istio-podinfo\") pod \"e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v\" (UID: \"4248f1a1-17bb-439c-80e5-8952de6299eb\") " pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v"
Oct 30 13:48:41.022941 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:41.022853    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4248f1a1-17bb-439c-80e5-8952de6299eb-istio-token\") pod \"e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v\" (UID: \"4248f1a1-17bb-439c-80e5-8952de6299eb\") " pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v"
Oct 30 13:48:41.111721 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:41.111629    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v"
Oct 30 13:48:41.455494 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:48:41.455457    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4248f1a1_17bb_439c_80e5_8952de6299eb.slice/crio-345c47f7f87892c5b64275340275810c9916c7e37fff9f5031e62cc36c26212e WatchSource:0}: Error finding container 345c47f7f87892c5b64275340275810c9916c7e37fff9f5031e62cc36c26212e: Status 404 returned error can't find the container with id 345c47f7f87892c5b64275340275810c9916c7e37fff9f5031e62cc36c26212e
Oct 30 13:48:41.456107 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:41.456079    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v"]
Oct 30 13:48:41.459535 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:41.459474    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:48:41.459825 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:41.459803    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:48:41.459999 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:41.459978    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:48:42.169273 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:42.169231    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v" event={"ID":"4248f1a1-17bb-439c-80e5-8952de6299eb","Type":"ContainerStarted","Data":"fb026cb86cadcf995e83773f4ace93a70ae80e522d3f5db954cae066427f38cf"}
Oct 30 13:48:42.169780 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:42.169294    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v" event={"ID":"4248f1a1-17bb-439c-80e5-8952de6299eb","Type":"ContainerStarted","Data":"345c47f7f87892c5b64275340275810c9916c7e37fff9f5031e62cc36c26212e"}
Oct 30 13:48:43.112754 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:43.112716    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v"
Oct 30 13:48:43.119943 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:43.119904    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v"
Oct 30 13:48:43.179407 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:43.179371    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v"
Oct 30 13:48:43.180642 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:43.180615    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v"
Oct 30 13:48:43.181158 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:43.181092    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v" podStartSLOduration=3.181077303 podStartE2EDuration="3.181077303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:48:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:42.239733434 +0000 UTC m=+2741.387000674" watchObservedRunningTime="2026-10-30 13:48:43.181077303 +0000 UTC m=+2742.328344544"
Oct 30 13:48:44.871368 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:44.871329    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7qmdwx"]
Oct 30 13:48:49.872313 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:49.872254    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7qmdwx" podUID="a247c1fd-0e91-4de7-aba7-465667dbd736" containerName="payload-processing" containerID="cri-o://a1c65fb6086ffb22767385d2d1932dbd0eb29fd1f40fd111ff43e48e5ebb5464" gracePeriod=25
Oct 30 13:48:50.174260 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:50.174225    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7qmdwx"
Oct 30 13:48:50.292221 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:50.292188    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a247c1fd-0e91-4de7-aba7-465667dbd736-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a247c1fd-0e91-4de7-aba7-465667dbd736-provider-credentials\") pod \"a247c1fd-0e91-4de7-aba7-465667dbd736\" (UID: \"a247c1fd-0e91-4de7-aba7-465667dbd736\") "
Oct 30 13:48:50.292421 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:50.292307    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a247c1fd-0e91-4de7-aba7-465667dbd736-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a247c1fd-0e91-4de7-aba7-465667dbd736-routing-overlay-volume\") pod \"a247c1fd-0e91-4de7-aba7-465667dbd736\" (UID: \"a247c1fd-0e91-4de7-aba7-465667dbd736\") "
Oct 30 13:48:50.292495 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:50.292451    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a247c1fd-0e91-4de7-aba7-465667dbd736-kube-api-access-7nhxf\" (UniqueName: \"kubernetes.io/projected/a247c1fd-0e91-4de7-aba7-465667dbd736-kube-api-access-7nhxf\") pod \"a247c1fd-0e91-4de7-aba7-465667dbd736\" (UID: \"a247c1fd-0e91-4de7-aba7-465667dbd736\") "
Oct 30 13:48:50.292569 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:50.292495    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a247c1fd-0e91-4de7-aba7-465667dbd736-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a247c1fd-0e91-4de7-aba7-465667dbd736-plugins-config-volume\") pod \"a247c1fd-0e91-4de7-aba7-465667dbd736\" (UID: \"a247c1fd-0e91-4de7-aba7-465667dbd736\") "
Oct 30 13:48:50.292830 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:50.292718    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a247c1fd-0e91-4de7-aba7-465667dbd736-routing-overlay-volume" pod "a247c1fd-0e91-4de7-aba7-465667dbd736" (UID: "a247c1fd-0e91-4de7-aba7-465667dbd736"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:48:50.292955 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:50.292922    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a247c1fd-0e91-4de7-aba7-465667dbd736-plugins-config-volume" pod "a247c1fd-0e91-4de7-aba7-465667dbd736" (UID: "a247c1fd-0e91-4de7-aba7-465667dbd736"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:48:50.293085 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:50.293069    5093 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a247c1fd-0e91-4de7-aba7-465667dbd736-routing-overlay-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:48:50.293167 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:50.293087    5093 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a247c1fd-0e91-4de7-aba7-465667dbd736-plugins-config-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:48:50.294557 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:50.294537    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a247c1fd-0e91-4de7-aba7-465667dbd736-kube-api-access-7nhxf" pod "a247c1fd-0e91-4de7-aba7-465667dbd736" (UID: "a247c1fd-0e91-4de7-aba7-465667dbd736"). InnerVolumeSpecName "kube-api-access-7nhxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:48:50.294660 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:50.294575    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a247c1fd-0e91-4de7-aba7-465667dbd736-provider-credentials" pod "a247c1fd-0e91-4de7-aba7-465667dbd736" (UID: "a247c1fd-0e91-4de7-aba7-465667dbd736"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:48:50.395101 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:50.395012    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nhxf\" (UniqueName: \"kubernetes.io/projected/a247c1fd-0e91-4de7-aba7-465667dbd736-kube-api-access-7nhxf\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:48:50.395101 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:50.395046    5093 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a247c1fd-0e91-4de7-aba7-465667dbd736-provider-credentials\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:48:50.475366 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:50.475325    5093 generic.go:356] "Generic (PLEG): container finished" podID="a247c1fd-0e91-4de7-aba7-465667dbd736" containerID="a1c65fb6086ffb22767385d2d1932dbd0eb29fd1f40fd111ff43e48e5ebb5464" exitCode=0
Oct 30 13:48:50.475366 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:50.475368    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7qmdwx"
Oct 30 13:48:50.475631 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:50.475402    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7qmdwx" event={"ID":"a247c1fd-0e91-4de7-aba7-465667dbd736","Type":"ContainerDied","Data":"a1c65fb6086ffb22767385d2d1932dbd0eb29fd1f40fd111ff43e48e5ebb5464"}
Oct 30 13:48:50.475631 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:50.475457    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7qmdwx" event={"ID":"a247c1fd-0e91-4de7-aba7-465667dbd736","Type":"ContainerDied","Data":"5eedb95b03deceedf36ea1dd99ae8e68703b266f9c4ccc489ef8c58f86f1e2a4"}
Oct 30 13:48:50.475631 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:50.475483    5093 scope.go:122] "RemoveContainer" containerID="a1c65fb6086ffb22767385d2d1932dbd0eb29fd1f40fd111ff43e48e5ebb5464"
Oct 30 13:48:50.487792 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:50.487773    5093 scope.go:122] "RemoveContainer" containerID="a1c65fb6086ffb22767385d2d1932dbd0eb29fd1f40fd111ff43e48e5ebb5464"
Oct 30 13:48:50.488088 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:48:50.488064    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1c65fb6086ffb22767385d2d1932dbd0eb29fd1f40fd111ff43e48e5ebb5464\": container with ID starting with a1c65fb6086ffb22767385d2d1932dbd0eb29fd1f40fd111ff43e48e5ebb5464 not found: ID does not exist" containerID="a1c65fb6086ffb22767385d2d1932dbd0eb29fd1f40fd111ff43e48e5ebb5464"
Oct 30 13:48:50.488149 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:50.488096    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1c65fb6086ffb22767385d2d1932dbd0eb29fd1f40fd111ff43e48e5ebb5464"} err="failed to get container status \"a1c65fb6086ffb22767385d2d1932dbd0eb29fd1f40fd111ff43e48e5ebb5464\": rpc error: code = NotFound desc = could not find container \"a1c65fb6086ffb22767385d2d1932dbd0eb29fd1f40fd111ff43e48e5ebb5464\": container with ID starting with a1c65fb6086ffb22767385d2d1932dbd0eb29fd1f40fd111ff43e48e5ebb5464 not found: ID does not exist"
Oct 30 13:48:50.534178 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:50.534119    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7qmdwx"]
Oct 30 13:48:50.548398 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:50.548349    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7qmdwx"]
Oct 30 13:48:51.001269 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:48:51.001234    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a247c1fd-0e91-4de7-aba7-465667dbd736" path="/var/lib/kubelet/pods/a247c1fd-0e91-4de7-aba7-465667dbd736/volumes"
Oct 30 13:48:51.169413 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:48:51.169373    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:49:14.157702 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:14.157659    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5494c9-684fdf5d4b-dp78d"]
Oct 30 13:49:14.158556 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:14.158538    5093 cpu_manager.go:414] "containerMap: removing container" podUID="a247c1fd-0e91-4de7-aba7-465667dbd736" containerName="payload-processing"
Oct 30 13:49:14.158613 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:14.158558    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a247c1fd-0e91-4de7-aba7-465667dbd736" containerName="payload-processing"
Oct 30 13:49:14.158714 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:14.158704    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="a247c1fd-0e91-4de7-aba7-465667dbd736" containerName="payload-processing"
Oct 30 13:49:14.164043 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:14.164022    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5494c9-684fdf5d4b-dp78d"
Oct 30 13:49:14.181202 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:14.181168    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5494c9-684fdf5d4b-dp78d"]
Oct 30 13:49:14.198665 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:14.188312    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-5494c9-serving-cert\""
Oct 30 13:49:14.198665 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:14.198146    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8aaca537-18ae-4b3a-9da2-3c834d740100-maas-api-tls\") pod \"maas-api-e2e-inf-a-5494c9-684fdf5d4b-dp78d\" (UID: \"8aaca537-18ae-4b3a-9da2-3c834d740100\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5494c9-684fdf5d4b-dp78d"
Oct 30 13:49:14.198665 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:14.198211    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8aaca537-18ae-4b3a-9da2-3c834d740100-metrics-certs\") pod \"maas-api-e2e-inf-a-5494c9-684fdf5d4b-dp78d\" (UID: \"8aaca537-18ae-4b3a-9da2-3c834d740100\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5494c9-684fdf5d4b-dp78d"
Oct 30 13:49:14.198665 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:14.198426    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aaca537-18ae-4b3a-9da2-3c834d740100-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5494c9-684fdf5d4b-dp78d\" (UID: \"8aaca537-18ae-4b3a-9da2-3c834d740100\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5494c9-684fdf5d4b-dp78d"
Oct 30 13:49:14.198665 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:14.198509    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlhbq\" (UniqueName: \"kubernetes.io/projected/8aaca537-18ae-4b3a-9da2-3c834d740100-kube-api-access-tlhbq\") pod \"maas-api-e2e-inf-a-5494c9-684fdf5d4b-dp78d\" (UID: \"8aaca537-18ae-4b3a-9da2-3c834d740100\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5494c9-684fdf5d4b-dp78d"
Oct 30 13:49:14.299751 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:14.299707    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8aaca537-18ae-4b3a-9da2-3c834d740100-maas-api-tls\") pod \"maas-api-e2e-inf-a-5494c9-684fdf5d4b-dp78d\" (UID: \"8aaca537-18ae-4b3a-9da2-3c834d740100\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5494c9-684fdf5d4b-dp78d"
Oct 30 13:49:14.299976 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:14.299770    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8aaca537-18ae-4b3a-9da2-3c834d740100-metrics-certs\") pod \"maas-api-e2e-inf-a-5494c9-684fdf5d4b-dp78d\" (UID: \"8aaca537-18ae-4b3a-9da2-3c834d740100\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5494c9-684fdf5d4b-dp78d"
Oct 30 13:49:14.299976 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:49:14.299867    5093 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-5494c9-serving-cert: secret "maas-api-e2e-inf-a-5494c9-serving-cert" not found
Oct 30 13:49:14.299976 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:14.299922    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aaca537-18ae-4b3a-9da2-3c834d740100-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5494c9-684fdf5d4b-dp78d\" (UID: \"8aaca537-18ae-4b3a-9da2-3c834d740100\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5494c9-684fdf5d4b-dp78d"
Oct 30 13:49:14.299976 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:49:14.299946    5093 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8aaca537-18ae-4b3a-9da2-3c834d740100-maas-api-tls podName:8aaca537-18ae-4b3a-9da2-3c834d740100 nodeName:}" failed. No retries permitted until 2026-10-30 13:49:14.799924612 +0000 UTC m=+2773.947191836 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8aaca537-18ae-4b3a-9da2-3c834d740100-maas-api-tls") pod "maas-api-e2e-inf-a-5494c9-684fdf5d4b-dp78d" (UID: "8aaca537-18ae-4b3a-9da2-3c834d740100") : secret "maas-api-e2e-inf-a-5494c9-serving-cert" not found
Oct 30 13:49:14.300246 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:14.300040    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlhbq\" (UniqueName: \"kubernetes.io/projected/8aaca537-18ae-4b3a-9da2-3c834d740100-kube-api-access-tlhbq\") pod \"maas-api-e2e-inf-a-5494c9-684fdf5d4b-dp78d\" (UID: \"8aaca537-18ae-4b3a-9da2-3c834d740100\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5494c9-684fdf5d4b-dp78d"
Oct 30 13:49:14.300690 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:14.300665    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aaca537-18ae-4b3a-9da2-3c834d740100-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5494c9-684fdf5d4b-dp78d\" (UID: \"8aaca537-18ae-4b3a-9da2-3c834d740100\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5494c9-684fdf5d4b-dp78d"
Oct 30 13:49:14.302438 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:14.302412    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8aaca537-18ae-4b3a-9da2-3c834d740100-metrics-certs\") pod \"maas-api-e2e-inf-a-5494c9-684fdf5d4b-dp78d\" (UID: \"8aaca537-18ae-4b3a-9da2-3c834d740100\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5494c9-684fdf5d4b-dp78d"
Oct 30 13:49:14.321812 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:14.321774    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlhbq\" (UniqueName: \"kubernetes.io/projected/8aaca537-18ae-4b3a-9da2-3c834d740100-kube-api-access-tlhbq\") pod \"maas-api-e2e-inf-a-5494c9-684fdf5d4b-dp78d\" (UID: \"8aaca537-18ae-4b3a-9da2-3c834d740100\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5494c9-684fdf5d4b-dp78d"
Oct 30 13:49:14.809348 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:14.809307    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8aaca537-18ae-4b3a-9da2-3c834d740100-maas-api-tls\") pod \"maas-api-e2e-inf-a-5494c9-684fdf5d4b-dp78d\" (UID: \"8aaca537-18ae-4b3a-9da2-3c834d740100\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5494c9-684fdf5d4b-dp78d"
Oct 30 13:49:14.811920 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:14.811888    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8aaca537-18ae-4b3a-9da2-3c834d740100-maas-api-tls\") pod \"maas-api-e2e-inf-a-5494c9-684fdf5d4b-dp78d\" (UID: \"8aaca537-18ae-4b3a-9da2-3c834d740100\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5494c9-684fdf5d4b-dp78d"
Oct 30 13:49:15.081374 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:15.081287    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5494c9-684fdf5d4b-dp78d"
Oct 30 13:49:15.455767 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:15.455735    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5494c9-684fdf5d4b-dp78d"]
Oct 30 13:49:15.456921 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:49:15.456893    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aaca537_18ae_4b3a_9da2_3c834d740100.slice/crio-8c32921d0fd9a73f765855b7b84bfc66efc9811e6f96c883df6edb8915a5f212 WatchSource:0}: Error finding container 8c32921d0fd9a73f765855b7b84bfc66efc9811e6f96c883df6edb8915a5f212: Status 404 returned error can't find the container with id 8c32921d0fd9a73f765855b7b84bfc66efc9811e6f96c883df6edb8915a5f212
Oct 30 13:49:15.710376 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:15.710286    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5494c9-684fdf5d4b-dp78d" event={"ID":"8aaca537-18ae-4b3a-9da2-3c834d740100","Type":"ContainerStarted","Data":"8c32921d0fd9a73f765855b7b84bfc66efc9811e6f96c883df6edb8915a5f212"}
Oct 30 13:49:16.720476 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:16.720438    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5494c9-684fdf5d4b-dp78d" event={"ID":"8aaca537-18ae-4b3a-9da2-3c834d740100","Type":"ContainerStarted","Data":"0d7423d7719916d1b981a5f0ad33dd1a3cd7d7534dd9abc3529c24031720751a"}
Oct 30 13:49:16.720906 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:16.720501    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5494c9-684fdf5d4b-dp78d"
Oct 30 13:49:16.775694 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:16.775635    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5494c9-684fdf5d4b-dp78d" podStartSLOduration=2.304441778 podStartE2EDuration="2.7756203s" totalImagesPullingTime="471.178522ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:49:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:49:16.772278772 +0000 UTC m=+2775.919546014" watchObservedRunningTime="2026-10-30 13:49:16.7756203 +0000 UTC m=+2775.922887542"
Oct 30 13:49:21.170070 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:49:21.170026    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:49:22.738840 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:22.738806    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5494c9-684fdf5d4b-dp78d"
Oct 30 13:49:41.978380 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:41.978344    5093 patch_prober.go:29] interesting pod/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 30 13:49:41.978835 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:41.978401    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx" podUID="d474316d-631a-4b79-baaf-2e19fb16a211" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 30 13:49:42.016281 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:42.016241    5093 generic.go:356] "Generic (PLEG): container finished" podID="d474316d-631a-4b79-baaf-2e19fb16a211" containerID="ef9cc275b940404c3b166458727f4ead429a04c9b6dc15604b44792bf5a197ad" exitCode=0
Oct 30 13:49:42.016467 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:42.016316    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx" event={"ID":"d474316d-631a-4b79-baaf-2e19fb16a211","Type":"ContainerDied","Data":"ef9cc275b940404c3b166458727f4ead429a04c9b6dc15604b44792bf5a197ad"}
Oct 30 13:49:42.017002 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:42.016982    5093 scope.go:122] "RemoveContainer" containerID="ef9cc275b940404c3b166458727f4ead429a04c9b6dc15604b44792bf5a197ad"
Oct 30 13:49:42.019158 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:42.019109    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:49:42.019279 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:42.019218    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:49:42.019327 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:42.019278    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:49:42.291570 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:42.291482    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-extmod-shr-user-d4ef2e/e2e-usage-sim-f3a8c8-8576b7fd6b-kdhdz"]
Oct 30 13:49:42.296590 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:42.296563    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-d4ef2e/e2e-usage-sim-f3a8c8-8576b7fd6b-kdhdz"
Oct 30 13:49:42.312262 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:42.312227    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-d4ef2e\"/\"kube-root-ca.crt\""
Oct 30 13:49:42.312431 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:42.312260    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-d4ef2e\"/\"openshift-service-ca.crt\""
Oct 30 13:49:42.312431 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:42.312345    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-d4ef2e\"/\"e2e-usage-sim-f3a8c8-cert\""
Oct 30 13:49:42.312431 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:42.312359    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-d4ef2e\"/\"default-dockercfg-qqtg6\""
Oct 30 13:49:42.312643 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:42.312618    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-d4ef2e/e2e-usage-sim-f3a8c8-8576b7fd6b-kdhdz"]
Oct 30 13:49:42.345157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:42.342915    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmsvt\" (UniqueName: \"kubernetes.io/projected/73976ccb-1e12-40ef-bf4b-6c5deeda3b88-kube-api-access-bmsvt\") pod \"e2e-usage-sim-f3a8c8-8576b7fd6b-kdhdz\" (UID: \"73976ccb-1e12-40ef-bf4b-6c5deeda3b88\") " pod="e2e-extmod-shr-user-d4ef2e/e2e-usage-sim-f3a8c8-8576b7fd6b-kdhdz"
Oct 30 13:49:42.345157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:42.343029    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/73976ccb-1e12-40ef-bf4b-6c5deeda3b88-tls\") pod \"e2e-usage-sim-f3a8c8-8576b7fd6b-kdhdz\" (UID: \"73976ccb-1e12-40ef-bf4b-6c5deeda3b88\") " pod="e2e-extmod-shr-user-d4ef2e/e2e-usage-sim-f3a8c8-8576b7fd6b-kdhdz"
Oct 30 13:49:42.445047 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:42.445010    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/73976ccb-1e12-40ef-bf4b-6c5deeda3b88-tls\") pod \"e2e-usage-sim-f3a8c8-8576b7fd6b-kdhdz\" (UID: \"73976ccb-1e12-40ef-bf4b-6c5deeda3b88\") " pod="e2e-extmod-shr-user-d4ef2e/e2e-usage-sim-f3a8c8-8576b7fd6b-kdhdz"
Oct 30 13:49:42.445361 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:42.445342    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmsvt\" (UniqueName: \"kubernetes.io/projected/73976ccb-1e12-40ef-bf4b-6c5deeda3b88-kube-api-access-bmsvt\") pod \"e2e-usage-sim-f3a8c8-8576b7fd6b-kdhdz\" (UID: \"73976ccb-1e12-40ef-bf4b-6c5deeda3b88\") " pod="e2e-extmod-shr-user-d4ef2e/e2e-usage-sim-f3a8c8-8576b7fd6b-kdhdz"
Oct 30 13:49:42.447698 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:42.447673    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/73976ccb-1e12-40ef-bf4b-6c5deeda3b88-tls\") pod \"e2e-usage-sim-f3a8c8-8576b7fd6b-kdhdz\" (UID: \"73976ccb-1e12-40ef-bf4b-6c5deeda3b88\") " pod="e2e-extmod-shr-user-d4ef2e/e2e-usage-sim-f3a8c8-8576b7fd6b-kdhdz"
Oct 30 13:49:42.465903 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:42.465867    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmsvt\" (UniqueName: \"kubernetes.io/projected/73976ccb-1e12-40ef-bf4b-6c5deeda3b88-kube-api-access-bmsvt\") pod \"e2e-usage-sim-f3a8c8-8576b7fd6b-kdhdz\" (UID: \"73976ccb-1e12-40ef-bf4b-6c5deeda3b88\") " pod="e2e-extmod-shr-user-d4ef2e/e2e-usage-sim-f3a8c8-8576b7fd6b-kdhdz"
Oct 30 13:49:42.613870 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:42.613778    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-d4ef2e/e2e-usage-sim-f3a8c8-8576b7fd6b-kdhdz"
Oct 30 13:49:42.975093 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:42.975045    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-d4ef2e/e2e-usage-sim-f3a8c8-8576b7fd6b-kdhdz"]
Oct 30 13:49:43.049151 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:43.047444    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-d4ef2e/e2e-usage-sim-f3a8c8-8576b7fd6b-kdhdz" event={"ID":"73976ccb-1e12-40ef-bf4b-6c5deeda3b88","Type":"ContainerStarted","Data":"183c935559916ea624f6f807f35ce1d34ecbc0432cbd0fa2ef671db0a77e7704"}
Oct 30 13:49:43.065058 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:43.065018    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx" event={"ID":"d474316d-631a-4b79-baaf-2e19fb16a211","Type":"ContainerStarted","Data":"b979fd59d3b2a04520ca52b9c57af7f10db30c394df8ba2a4d993b8fdab4e8d3"}
Oct 30 13:49:43.417495 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:43.417407    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx"
Oct 30 13:49:43.417665 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:43.417622    5093 patch_prober.go:29] interesting pod/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 30 13:49:43.417717 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:43.417669    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx" podUID="d474316d-631a-4b79-baaf-2e19fb16a211" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 30 13:49:44.128327 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:44.128287    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-d4ef2e/e2e-usage-sim-f3a8c8-8576b7fd6b-kdhdz" event={"ID":"73976ccb-1e12-40ef-bf4b-6c5deeda3b88","Type":"ContainerStarted","Data":"3d46ad85b1a67a5010c1ad0f57cf70899bcee151fa7e817abfc26f0f18c2a26b"}
Oct 30 13:49:44.160158 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:44.129201    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-extmod-shr-user-d4ef2e/e2e-usage-sim-f3a8c8-8576b7fd6b-kdhdz"
Oct 30 13:49:44.193535 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:44.193446    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-extmod-shr-user-d4ef2e/e2e-usage-sim-f3a8c8-8576b7fd6b-kdhdz" podStartSLOduration=2.193423007 podStartE2EDuration="2.193423007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:49:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:49:44.17851905 +0000 UTC m=+2803.325786291" watchObservedRunningTime="2026-10-30 13:49:44.193423007 +0000 UTC m=+2803.340690249"
Oct 30 13:49:44.198847 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:44.198815    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-extmod-shr-user-d4ef2e/e2e-usage-sim-f3a8c8-8576b7fd6b-kdhdz"
Oct 30 13:49:44.417303 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:44.417201    5093 patch_prober.go:29] interesting pod/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 30 13:49:44.417303 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:44.417263    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx" podUID="d474316d-631a-4b79-baaf-2e19fb16a211" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 30 13:49:45.417214 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:45.417175    5093 patch_prober.go:29] interesting pod/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 30 13:49:45.417644 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:45.417238    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx" podUID="d474316d-631a-4b79-baaf-2e19fb16a211" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 30 13:49:46.417817 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:46.417783    5093 patch_prober.go:29] interesting pod/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 30 13:49:46.418304 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:46.417839    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx" podUID="d474316d-631a-4b79-baaf-2e19fb16a211" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 30 13:49:47.417499 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:47.417459    5093 patch_prober.go:29] interesting pod/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 30 13:49:47.417700 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:47.417519    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx" podUID="d474316d-631a-4b79-baaf-2e19fb16a211" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 30 13:49:48.417701 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:48.417662    5093 patch_prober.go:29] interesting pod/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 30 13:49:48.418192 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:48.417719    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx" podUID="d474316d-631a-4b79-baaf-2e19fb16a211" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 30 13:49:49.457152 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:49.448446    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx"
Oct 30 13:49:49.457152 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:49.450145    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx"
Oct 30 13:49:49.476154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:49:49.473421    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx"
Oct 30 13:49:51.168574 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:49:51.168529    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:50:06.305271 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:06.305231    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6cd7tcwgw"]
Oct 30 13:50:07.183695 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:07.183658    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6558x68zp"]
Oct 30 13:50:11.307776 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:11.307623    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6cd7tcwgw" podUID="6f6422f2-f1b6-433f-93a3-52edd828c682" containerName="payload-processing" containerID="cri-o://362173a912f5f1fcb25e3b1956c812a35c20fbbdb44a5add473ffde334ee99a0" gracePeriod=25
Oct 30 13:50:11.784246 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:11.784209    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6cd7tcwgw"
Oct 30 13:50:11.828154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:11.823082    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f6422f2-f1b6-433f-93a3-52edd828c682-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6f6422f2-f1b6-433f-93a3-52edd828c682-routing-overlay-volume\") pod \"6f6422f2-f1b6-433f-93a3-52edd828c682\" (UID: \"6f6422f2-f1b6-433f-93a3-52edd828c682\") "
Oct 30 13:50:11.828154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:11.823249    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f6422f2-f1b6-433f-93a3-52edd828c682-kube-api-access-vrwws\" (UniqueName: \"kubernetes.io/projected/6f6422f2-f1b6-433f-93a3-52edd828c682-kube-api-access-vrwws\") pod \"6f6422f2-f1b6-433f-93a3-52edd828c682\" (UID: \"6f6422f2-f1b6-433f-93a3-52edd828c682\") "
Oct 30 13:50:11.828154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:11.823319    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f6422f2-f1b6-433f-93a3-52edd828c682-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6f6422f2-f1b6-433f-93a3-52edd828c682-provider-credentials\") pod \"6f6422f2-f1b6-433f-93a3-52edd828c682\" (UID: \"6f6422f2-f1b6-433f-93a3-52edd828c682\") "
Oct 30 13:50:11.828154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:11.823407    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f6422f2-f1b6-433f-93a3-52edd828c682-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f6422f2-f1b6-433f-93a3-52edd828c682-plugins-config-volume\") pod \"6f6422f2-f1b6-433f-93a3-52edd828c682\" (UID: \"6f6422f2-f1b6-433f-93a3-52edd828c682\") "
Oct 30 13:50:11.853153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:11.836528    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f6422f2-f1b6-433f-93a3-52edd828c682-routing-overlay-volume" pod "6f6422f2-f1b6-433f-93a3-52edd828c682" (UID: "6f6422f2-f1b6-433f-93a3-52edd828c682"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:50:11.853153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:11.852682    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f6422f2-f1b6-433f-93a3-52edd828c682-plugins-config-volume" pod "6f6422f2-f1b6-433f-93a3-52edd828c682" (UID: "6f6422f2-f1b6-433f-93a3-52edd828c682"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:50:11.882348 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:11.881269    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f6422f2-f1b6-433f-93a3-52edd828c682-provider-credentials" pod "6f6422f2-f1b6-433f-93a3-52edd828c682" (UID: "6f6422f2-f1b6-433f-93a3-52edd828c682"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:50:11.935153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:11.925482    5093 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6f6422f2-f1b6-433f-93a3-52edd828c682-provider-credentials\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:50:11.935153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:11.925525    5093 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f6422f2-f1b6-433f-93a3-52edd828c682-plugins-config-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:50:11.935153 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:11.925539    5093 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6f6422f2-f1b6-433f-93a3-52edd828c682-routing-overlay-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:50:11.974152 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:11.957262    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f6422f2-f1b6-433f-93a3-52edd828c682-kube-api-access-vrwws" pod "6f6422f2-f1b6-433f-93a3-52edd828c682" (UID: "6f6422f2-f1b6-433f-93a3-52edd828c682"). InnerVolumeSpecName "kube-api-access-vrwws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:50:12.039156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:12.035617    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrwws\" (UniqueName: \"kubernetes.io/projected/6f6422f2-f1b6-433f-93a3-52edd828c682-kube-api-access-vrwws\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:50:12.190211 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:12.184942    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6558x68zp" podUID="a4e01c3f-d5a5-47c8-8d75-33883b7b3d74" containerName="payload-processing" containerID="cri-o://cd50346345055674a36da0abdfbfaa80998aaf3163faa889a1e9d3958a70edef" gracePeriod=25
Oct 30 13:50:12.526912 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:12.525324    5093 generic.go:356] "Generic (PLEG): container finished" podID="6f6422f2-f1b6-433f-93a3-52edd828c682" containerID="362173a912f5f1fcb25e3b1956c812a35c20fbbdb44a5add473ffde334ee99a0" exitCode=0
Oct 30 13:50:12.526912 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:12.525660    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6cd7tcwgw" event={"ID":"6f6422f2-f1b6-433f-93a3-52edd828c682","Type":"ContainerDied","Data":"362173a912f5f1fcb25e3b1956c812a35c20fbbdb44a5add473ffde334ee99a0"}
Oct 30 13:50:12.526912 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:12.525787    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6cd7tcwgw" event={"ID":"6f6422f2-f1b6-433f-93a3-52edd828c682","Type":"ContainerDied","Data":"04b57a9789e0a0ba65578e2996c2e94b32c8d704d277766cc05cbe6e363c5f2b"}
Oct 30 13:50:12.526912 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:12.525822    5093 scope.go:122] "RemoveContainer" containerID="362173a912f5f1fcb25e3b1956c812a35c20fbbdb44a5add473ffde334ee99a0"
Oct 30 13:50:12.529199 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:12.529173    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6cd7tcwgw"
Oct 30 13:50:12.544878 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:12.544649    5093 generic.go:356] "Generic (PLEG): container finished" podID="a4e01c3f-d5a5-47c8-8d75-33883b7b3d74" containerID="cd50346345055674a36da0abdfbfaa80998aaf3163faa889a1e9d3958a70edef" exitCode=0
Oct 30 13:50:12.544878 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:12.544709    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6558x68zp" event={"ID":"a4e01c3f-d5a5-47c8-8d75-33883b7b3d74","Type":"ContainerDied","Data":"cd50346345055674a36da0abdfbfaa80998aaf3163faa889a1e9d3958a70edef"}
Oct 30 13:50:12.640170 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:12.640003    5093 scope.go:122] "RemoveContainer" containerID="362173a912f5f1fcb25e3b1956c812a35c20fbbdb44a5add473ffde334ee99a0"
Oct 30 13:50:12.641010 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:50:12.640974    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"362173a912f5f1fcb25e3b1956c812a35c20fbbdb44a5add473ffde334ee99a0\": container with ID starting with 362173a912f5f1fcb25e3b1956c812a35c20fbbdb44a5add473ffde334ee99a0 not found: ID does not exist" containerID="362173a912f5f1fcb25e3b1956c812a35c20fbbdb44a5add473ffde334ee99a0"
Oct 30 13:50:12.641217 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:12.641185    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"362173a912f5f1fcb25e3b1956c812a35c20fbbdb44a5add473ffde334ee99a0"} err="failed to get container status \"362173a912f5f1fcb25e3b1956c812a35c20fbbdb44a5add473ffde334ee99a0\": rpc error: code = NotFound desc = could not find container \"362173a912f5f1fcb25e3b1956c812a35c20fbbdb44a5add473ffde334ee99a0\": container with ID starting with 362173a912f5f1fcb25e3b1956c812a35c20fbbdb44a5add473ffde334ee99a0 not found: ID does not exist"
Oct 30 13:50:12.669083 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:12.669046    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6cd7tcwgw"]
Oct 30 13:50:12.681378 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:12.681338    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6cd7tcwgw"]
Oct 30 13:50:12.761861 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:12.761830    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6558x68zp"
Oct 30 13:50:12.765976 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:12.765945    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4e01c3f-d5a5-47c8-8d75-33883b7b3d74-kube-api-access-k5xxx\" (UniqueName: \"kubernetes.io/projected/a4e01c3f-d5a5-47c8-8d75-33883b7b3d74-kube-api-access-k5xxx\") pod \"a4e01c3f-d5a5-47c8-8d75-33883b7b3d74\" (UID: \"a4e01c3f-d5a5-47c8-8d75-33883b7b3d74\") "
Oct 30 13:50:12.766260 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:12.765993    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4e01c3f-d5a5-47c8-8d75-33883b7b3d74-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a4e01c3f-d5a5-47c8-8d75-33883b7b3d74-routing-overlay-volume\") pod \"a4e01c3f-d5a5-47c8-8d75-33883b7b3d74\" (UID: \"a4e01c3f-d5a5-47c8-8d75-33883b7b3d74\") "
Oct 30 13:50:12.766260 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:12.766155    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4e01c3f-d5a5-47c8-8d75-33883b7b3d74-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a4e01c3f-d5a5-47c8-8d75-33883b7b3d74-provider-credentials\") pod \"a4e01c3f-d5a5-47c8-8d75-33883b7b3d74\" (UID: \"a4e01c3f-d5a5-47c8-8d75-33883b7b3d74\") "
Oct 30 13:50:12.766260 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:12.766234    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4e01c3f-d5a5-47c8-8d75-33883b7b3d74-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4e01c3f-d5a5-47c8-8d75-33883b7b3d74-plugins-config-volume\") pod \"a4e01c3f-d5a5-47c8-8d75-33883b7b3d74\" (UID: \"a4e01c3f-d5a5-47c8-8d75-33883b7b3d74\") "
Oct 30 13:50:12.769978 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:12.766474    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4e01c3f-d5a5-47c8-8d75-33883b7b3d74-routing-overlay-volume" pod "a4e01c3f-d5a5-47c8-8d75-33883b7b3d74" (UID: "a4e01c3f-d5a5-47c8-8d75-33883b7b3d74"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:50:12.769978 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:12.766770    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4e01c3f-d5a5-47c8-8d75-33883b7b3d74-plugins-config-volume" pod "a4e01c3f-d5a5-47c8-8d75-33883b7b3d74" (UID: "a4e01c3f-d5a5-47c8-8d75-33883b7b3d74"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:50:12.769978 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:12.769166    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4e01c3f-d5a5-47c8-8d75-33883b7b3d74-provider-credentials" pod "a4e01c3f-d5a5-47c8-8d75-33883b7b3d74" (UID: "a4e01c3f-d5a5-47c8-8d75-33883b7b3d74"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:50:12.769978 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:12.769210    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4e01c3f-d5a5-47c8-8d75-33883b7b3d74-kube-api-access-k5xxx" pod "a4e01c3f-d5a5-47c8-8d75-33883b7b3d74" (UID: "a4e01c3f-d5a5-47c8-8d75-33883b7b3d74"). InnerVolumeSpecName "kube-api-access-k5xxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:50:12.869297 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:12.869251    5093 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a4e01c3f-d5a5-47c8-8d75-33883b7b3d74-provider-credentials\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:50:12.869297 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:12.869299    5093 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4e01c3f-d5a5-47c8-8d75-33883b7b3d74-plugins-config-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:50:12.869524 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:12.869316    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5xxx\" (UniqueName: \"kubernetes.io/projected/a4e01c3f-d5a5-47c8-8d75-33883b7b3d74-kube-api-access-k5xxx\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:50:12.869524 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:12.869330    5093 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a4e01c3f-d5a5-47c8-8d75-33883b7b3d74-routing-overlay-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:50:13.001894 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:13.001858    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f6422f2-f1b6-433f-93a3-52edd828c682" path="/var/lib/kubelet/pods/6f6422f2-f1b6-433f-93a3-52edd828c682/volumes"
Oct 30 13:50:13.555117 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:13.555083    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6558x68zp"
Oct 30 13:50:13.555612 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:13.555117    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6558x68zp" event={"ID":"a4e01c3f-d5a5-47c8-8d75-33883b7b3d74","Type":"ContainerDied","Data":"6207aaaa4aeda0f6fe080b5bc11704809581dcb9f7d521c9c04184bd88d206ff"}
Oct 30 13:50:13.555612 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:13.555197    5093 scope.go:122] "RemoveContainer" containerID="cd50346345055674a36da0abdfbfaa80998aaf3163faa889a1e9d3958a70edef"
Oct 30 13:50:13.637844 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:13.637809    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6558x68zp"]
Oct 30 13:50:13.647900 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:13.647867    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6558x68zp"]
Oct 30 13:50:15.000908 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:15.000869    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4e01c3f-d5a5-47c8-8d75-33883b7b3d74" path="/var/lib/kubelet/pods/a4e01c3f-d5a5-47c8-8d75-33883b7b3d74/volumes"
Oct 30 13:50:15.192152 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:15.191714    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2"]
Oct 30 13:50:15.193054 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:15.193023    5093 cpu_manager.go:414] "containerMap: removing container" podUID="a4e01c3f-d5a5-47c8-8d75-33883b7b3d74" containerName="payload-processing"
Oct 30 13:50:15.193054 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:15.193052    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4e01c3f-d5a5-47c8-8d75-33883b7b3d74" containerName="payload-processing"
Oct 30 13:50:15.193303 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:15.193087    5093 cpu_manager.go:414] "containerMap: removing container" podUID="6f6422f2-f1b6-433f-93a3-52edd828c682" containerName="payload-processing"
Oct 30 13:50:15.193303 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:15.193097    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f6422f2-f1b6-433f-93a3-52edd828c682" containerName="payload-processing"
Oct 30 13:50:15.193433 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:15.193308    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4e01c3f-d5a5-47c8-8d75-33883b7b3d74" containerName="payload-processing"
Oct 30 13:50:15.193433 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:15.193326    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f6422f2-f1b6-433f-93a3-52edd828c682" containerName="payload-processing"
Oct 30 13:50:15.202714 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:15.202684    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2"
Oct 30 13:50:15.219405 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:15.219377    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2"]
Oct 30 13:50:15.271653 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:15.271566    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-08c365-openshift-default-dockercfg-grjr6\""
Oct 30 13:50:15.307840 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:15.307798    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe9a708-c8aa-414b-80a8-fcd5c1142425-credential-socket\") pod \"e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2\" (UID: \"abe9a708-c8aa-414b-80a8-fcd5c1142425\") " pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2"
Oct 30 13:50:15.308114 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:15.308087    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abe9a708-c8aa-414b-80a8-fcd5c1142425-istio-token\") pod \"e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2\" (UID: \"abe9a708-c8aa-414b-80a8-fcd5c1142425\") " pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2"
Oct 30 13:50:15.308398 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:15.308377    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abe9a708-c8aa-414b-80a8-fcd5c1142425-istio-data\") pod \"e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2\" (UID: \"abe9a708-c8aa-414b-80a8-fcd5c1142425\") " pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2"
Oct 30 13:50:15.308670 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:15.308636    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abe9a708-c8aa-414b-80a8-fcd5c1142425-istio-podinfo\") pod \"e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2\" (UID: \"abe9a708-c8aa-414b-80a8-fcd5c1142425\") " pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2"
Oct 30 13:50:15.308949 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:15.308924    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abe9a708-c8aa-414b-80a8-fcd5c1142425-workload-certs\") pod \"e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2\" (UID: \"abe9a708-c8aa-414b-80a8-fcd5c1142425\") " pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2"
Oct 30 13:50:15.309316 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:15.309295    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abe9a708-c8aa-414b-80a8-fcd5c1142425-istio-envoy\") pod \"e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2\" (UID: \"abe9a708-c8aa-414b-80a8-fcd5c1142425\") " pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2"
Oct 30 13:50:15.309512 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:15.309494    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68w8w\" (UniqueName: \"kubernetes.io/projected/abe9a708-c8aa-414b-80a8-fcd5c1142425-kube-api-access-68w8w\") pod \"e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2\" (UID: \"abe9a708-c8aa-414b-80a8-fcd5c1142425\") " pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2"
Oct 30 13:50:15.309697 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:15.309680    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abe9a708-c8aa-414b-80a8-fcd5c1142425-istiod-ca-cert\") pod \"e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2\" (UID: \"abe9a708-c8aa-414b-80a8-fcd5c1142425\") " pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2"
Oct 30 13:50:15.309864 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:15.309849    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe9a708-c8aa-414b-80a8-fcd5c1142425-workload-socket\") pod \"e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2\" (UID: \"abe9a708-c8aa-414b-80a8-fcd5c1142425\") " pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2"
Oct 30 13:50:15.414344 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:15.414303    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abe9a708-c8aa-414b-80a8-fcd5c1142425-workload-certs\") pod \"e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2\" (UID: \"abe9a708-c8aa-414b-80a8-fcd5c1142425\") " pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2"
Oct 30 13:50:15.414935 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:15.414908    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abe9a708-c8aa-414b-80a8-fcd5c1142425-istio-envoy\") pod \"e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2\" (UID: \"abe9a708-c8aa-414b-80a8-fcd5c1142425\") " pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2"
Oct 30 13:50:15.415436 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:15.415415    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68w8w\" (UniqueName: \"kubernetes.io/projected/abe9a708-c8aa-414b-80a8-fcd5c1142425-kube-api-access-68w8w\") pod \"e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2\" (UID: \"abe9a708-c8aa-414b-80a8-fcd5c1142425\") " pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2"
Oct 30 13:50:15.415981 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:15.415959    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abe9a708-c8aa-414b-80a8-fcd5c1142425-istiod-ca-cert\") pod \"e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2\" (UID: \"abe9a708-c8aa-414b-80a8-fcd5c1142425\") " pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2"
Oct 30 13:50:15.416396 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:15.416378    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe9a708-c8aa-414b-80a8-fcd5c1142425-workload-socket\") pod \"e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2\" (UID: \"abe9a708-c8aa-414b-80a8-fcd5c1142425\") " pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2"
Oct 30 13:50:15.416843 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:15.416824    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe9a708-c8aa-414b-80a8-fcd5c1142425-credential-socket\") pod \"e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2\" (UID: \"abe9a708-c8aa-414b-80a8-fcd5c1142425\") " pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2"
Oct 30 13:50:15.416990 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:15.416974    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abe9a708-c8aa-414b-80a8-fcd5c1142425-istio-token\") pod \"e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2\" (UID: \"abe9a708-c8aa-414b-80a8-fcd5c1142425\") " pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2"
Oct 30 13:50:15.417168 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:15.417152    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abe9a708-c8aa-414b-80a8-fcd5c1142425-istio-data\") pod \"e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2\" (UID: \"abe9a708-c8aa-414b-80a8-fcd5c1142425\") " pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2"
Oct 30 13:50:15.417395 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:15.417379    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abe9a708-c8aa-414b-80a8-fcd5c1142425-istio-podinfo\") pod \"e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2\" (UID: \"abe9a708-c8aa-414b-80a8-fcd5c1142425\") " pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2"
Oct 30 13:50:15.428171 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:15.428118    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abe9a708-c8aa-414b-80a8-fcd5c1142425-istio-data\") pod \"e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2\" (UID: \"abe9a708-c8aa-414b-80a8-fcd5c1142425\") " pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2"
Oct 30 13:50:15.431109 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:15.429628    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe9a708-c8aa-414b-80a8-fcd5c1142425-credential-socket\") pod \"e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2\" (UID: \"abe9a708-c8aa-414b-80a8-fcd5c1142425\") " pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2"
Oct 30 13:50:15.431109 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:15.429812    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe9a708-c8aa-414b-80a8-fcd5c1142425-workload-socket\") pod \"e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2\" (UID: \"abe9a708-c8aa-414b-80a8-fcd5c1142425\") " pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2"
Oct 30 13:50:15.431109 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:15.430382    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abe9a708-c8aa-414b-80a8-fcd5c1142425-istiod-ca-cert\") pod \"e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2\" (UID: \"abe9a708-c8aa-414b-80a8-fcd5c1142425\") " pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2"
Oct 30 13:50:15.440161 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:15.439770    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abe9a708-c8aa-414b-80a8-fcd5c1142425-workload-certs\") pod \"e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2\" (UID: \"abe9a708-c8aa-414b-80a8-fcd5c1142425\") " pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2"
Oct 30 13:50:15.462163 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:15.459849    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68w8w\" (UniqueName: \"kubernetes.io/projected/abe9a708-c8aa-414b-80a8-fcd5c1142425-kube-api-access-68w8w\") pod \"e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2\" (UID: \"abe9a708-c8aa-414b-80a8-fcd5c1142425\") " pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2"
Oct 30 13:50:15.473166 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:15.471793    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abe9a708-c8aa-414b-80a8-fcd5c1142425-istio-token\") pod \"e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2\" (UID: \"abe9a708-c8aa-414b-80a8-fcd5c1142425\") " pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2"
Oct 30 13:50:15.499773 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:15.498891    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abe9a708-c8aa-414b-80a8-fcd5c1142425-istio-podinfo\") pod \"e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2\" (UID: \"abe9a708-c8aa-414b-80a8-fcd5c1142425\") " pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2"
Oct 30 13:50:15.499773 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:15.499358    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abe9a708-c8aa-414b-80a8-fcd5c1142425-istio-envoy\") pod \"e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2\" (UID: \"abe9a708-c8aa-414b-80a8-fcd5c1142425\") " pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2"
Oct 30 13:50:15.530204 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:15.529860    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2"
Oct 30 13:50:15.878275 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:15.878241    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2"]
Oct 30 13:50:15.879330 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:50:15.879293    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabe9a708_c8aa_414b_80a8_fcd5c1142425.slice/crio-65ff1d71723cd93f71becc63b06b74ab40c495a535f83440ea3cfc8b67aa3530 WatchSource:0}: Error finding container 65ff1d71723cd93f71becc63b06b74ab40c495a535f83440ea3cfc8b67aa3530: Status 404 returned error can't find the container with id 65ff1d71723cd93f71becc63b06b74ab40c495a535f83440ea3cfc8b67aa3530
Oct 30 13:50:15.882319 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:15.882285    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:50:15.882398 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:15.882354    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:50:15.882398 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:15.882387    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:50:16.585569 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:16.585528    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2" event={"ID":"abe9a708-c8aa-414b-80a8-fcd5c1142425","Type":"ContainerStarted","Data":"6813c626f4ef36b3ac1dedde6e29d5ae2cd0db66fed3061904338d971f29665f"}
Oct 30 13:50:16.586071 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:16.585622    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2" event={"ID":"abe9a708-c8aa-414b-80a8-fcd5c1142425","Type":"ContainerStarted","Data":"65ff1d71723cd93f71becc63b06b74ab40c495a535f83440ea3cfc8b67aa3530"}
Oct 30 13:50:17.539779 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:17.539743    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2"
Oct 30 13:50:17.548289 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:17.548257    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2"
Oct 30 13:50:17.594796 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:17.594760    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2"
Oct 30 13:50:17.595736 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:17.595711    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2"
Oct 30 13:50:17.622622 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:17.622562    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2" podStartSLOduration=2.622548181 podStartE2EDuration="2.622548181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:50:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:50:16.658690219 +0000 UTC m=+2835.805957460" watchObservedRunningTime="2026-10-30 13:50:17.622548181 +0000 UTC m=+2836.769815422"
Oct 30 13:50:21.377854 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:50:21.377800    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:50:28.072091 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:28.072055    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj"]
Oct 30 13:50:28.080279 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:28.080250    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj"
Oct 30 13:50:28.092122 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:28.092093    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj"]
Oct 30 13:50:28.097587 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:28.097533    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-88352d-openshift-default-dockercfg-b5flw\""
Oct 30 13:50:28.104440 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:28.104406    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1b0539d-aead-4e37-b4ba-d6de98ae472b-istio-data\") pod \"e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj\" (UID: \"f1b0539d-aead-4e37-b4ba-d6de98ae472b\") " pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj"
Oct 30 13:50:28.104594 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:28.104467    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1b0539d-aead-4e37-b4ba-d6de98ae472b-istio-envoy\") pod \"e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj\" (UID: \"f1b0539d-aead-4e37-b4ba-d6de98ae472b\") " pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj"
Oct 30 13:50:28.104594 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:28.104488    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1b0539d-aead-4e37-b4ba-d6de98ae472b-istio-token\") pod \"e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj\" (UID: \"f1b0539d-aead-4e37-b4ba-d6de98ae472b\") " pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj"
Oct 30 13:50:28.104594 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:28.104531    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ssf8\" (UniqueName: \"kubernetes.io/projected/f1b0539d-aead-4e37-b4ba-d6de98ae472b-kube-api-access-6ssf8\") pod \"e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj\" (UID: \"f1b0539d-aead-4e37-b4ba-d6de98ae472b\") " pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj"
Oct 30 13:50:28.104886 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:28.104864    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1b0539d-aead-4e37-b4ba-d6de98ae472b-istio-podinfo\") pod \"e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj\" (UID: \"f1b0539d-aead-4e37-b4ba-d6de98ae472b\") " pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj"
Oct 30 13:50:28.104989 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:28.104976    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1b0539d-aead-4e37-b4ba-d6de98ae472b-workload-certs\") pod \"e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj\" (UID: \"f1b0539d-aead-4e37-b4ba-d6de98ae472b\") " pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj"
Oct 30 13:50:28.105055 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:28.105002    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1b0539d-aead-4e37-b4ba-d6de98ae472b-workload-socket\") pod \"e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj\" (UID: \"f1b0539d-aead-4e37-b4ba-d6de98ae472b\") " pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj"
Oct 30 13:50:28.105232 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:28.105199    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1b0539d-aead-4e37-b4ba-d6de98ae472b-istiod-ca-cert\") pod \"e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj\" (UID: \"f1b0539d-aead-4e37-b4ba-d6de98ae472b\") " pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj"
Oct 30 13:50:28.105322 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:28.105261    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1b0539d-aead-4e37-b4ba-d6de98ae472b-credential-socket\") pod \"e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj\" (UID: \"f1b0539d-aead-4e37-b4ba-d6de98ae472b\") " pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj"
Oct 30 13:50:28.206334 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:28.206269    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1b0539d-aead-4e37-b4ba-d6de98ae472b-istiod-ca-cert\") pod \"e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj\" (UID: \"f1b0539d-aead-4e37-b4ba-d6de98ae472b\") " pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj"
Oct 30 13:50:28.206513 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:28.206411    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1b0539d-aead-4e37-b4ba-d6de98ae472b-credential-socket\") pod \"e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj\" (UID: \"f1b0539d-aead-4e37-b4ba-d6de98ae472b\") " pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj"
Oct 30 13:50:28.206600 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:28.206533    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1b0539d-aead-4e37-b4ba-d6de98ae472b-istio-data\") pod \"e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj\" (UID: \"f1b0539d-aead-4e37-b4ba-d6de98ae472b\") " pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj"
Oct 30 13:50:28.206600 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:28.206588    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1b0539d-aead-4e37-b4ba-d6de98ae472b-istio-envoy\") pod \"e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj\" (UID: \"f1b0539d-aead-4e37-b4ba-d6de98ae472b\") " pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj"
Oct 30 13:50:28.206715 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:28.206624    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1b0539d-aead-4e37-b4ba-d6de98ae472b-istio-token\") pod \"e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj\" (UID: \"f1b0539d-aead-4e37-b4ba-d6de98ae472b\") " pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj"
Oct 30 13:50:28.206715 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:28.206689    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ssf8\" (UniqueName: \"kubernetes.io/projected/f1b0539d-aead-4e37-b4ba-d6de98ae472b-kube-api-access-6ssf8\") pod \"e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj\" (UID: \"f1b0539d-aead-4e37-b4ba-d6de98ae472b\") " pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj"
Oct 30 13:50:28.206814 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:28.206756    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1b0539d-aead-4e37-b4ba-d6de98ae472b-istio-podinfo\") pod \"e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj\" (UID: \"f1b0539d-aead-4e37-b4ba-d6de98ae472b\") " pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj"
Oct 30 13:50:28.206874 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:28.206835    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1b0539d-aead-4e37-b4ba-d6de98ae472b-workload-certs\") pod \"e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj\" (UID: \"f1b0539d-aead-4e37-b4ba-d6de98ae472b\") " pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj"
Oct 30 13:50:28.206929 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:28.206870    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1b0539d-aead-4e37-b4ba-d6de98ae472b-workload-socket\") pod \"e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj\" (UID: \"f1b0539d-aead-4e37-b4ba-d6de98ae472b\") " pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj"
Oct 30 13:50:28.206989 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:28.206923    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1b0539d-aead-4e37-b4ba-d6de98ae472b-istiod-ca-cert\") pod \"e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj\" (UID: \"f1b0539d-aead-4e37-b4ba-d6de98ae472b\") " pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj"
Oct 30 13:50:28.207408 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:28.207375    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1b0539d-aead-4e37-b4ba-d6de98ae472b-workload-certs\") pod \"e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj\" (UID: \"f1b0539d-aead-4e37-b4ba-d6de98ae472b\") " pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj"
Oct 30 13:50:28.207408 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:28.207393    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1b0539d-aead-4e37-b4ba-d6de98ae472b-istio-data\") pod \"e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj\" (UID: \"f1b0539d-aead-4e37-b4ba-d6de98ae472b\") " pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj"
Oct 30 13:50:28.207644 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:28.207631    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1b0539d-aead-4e37-b4ba-d6de98ae472b-workload-socket\") pod \"e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj\" (UID: \"f1b0539d-aead-4e37-b4ba-d6de98ae472b\") " pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj"
Oct 30 13:50:28.208239 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:28.208204    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1b0539d-aead-4e37-b4ba-d6de98ae472b-credential-socket\") pod \"e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj\" (UID: \"f1b0539d-aead-4e37-b4ba-d6de98ae472b\") " pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj"
Oct 30 13:50:28.210672 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:28.210642    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1b0539d-aead-4e37-b4ba-d6de98ae472b-istio-podinfo\") pod \"e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj\" (UID: \"f1b0539d-aead-4e37-b4ba-d6de98ae472b\") " pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj"
Oct 30 13:50:28.210862 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:28.210844    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1b0539d-aead-4e37-b4ba-d6de98ae472b-istio-envoy\") pod \"e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj\" (UID: \"f1b0539d-aead-4e37-b4ba-d6de98ae472b\") " pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj"
Oct 30 13:50:28.287050 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:28.286974    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1b0539d-aead-4e37-b4ba-d6de98ae472b-istio-token\") pod \"e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj\" (UID: \"f1b0539d-aead-4e37-b4ba-d6de98ae472b\") " pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj"
Oct 30 13:50:28.304164 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:28.301864    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ssf8\" (UniqueName: \"kubernetes.io/projected/f1b0539d-aead-4e37-b4ba-d6de98ae472b-kube-api-access-6ssf8\") pod \"e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj\" (UID: \"f1b0539d-aead-4e37-b4ba-d6de98ae472b\") " pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj"
Oct 30 13:50:28.406906 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:28.406810    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj"
Oct 30 13:50:28.859901 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:50:28.859866    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1b0539d_aead_4e37_b4ba_d6de98ae472b.slice/crio-6e6679b43f256b62b539fb8096db94503c65f60cfc536154764a39e09127b628 WatchSource:0}: Error finding container 6e6679b43f256b62b539fb8096db94503c65f60cfc536154764a39e09127b628: Status 404 returned error can't find the container with id 6e6679b43f256b62b539fb8096db94503c65f60cfc536154764a39e09127b628
Oct 30 13:50:28.860648 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:28.860619    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj"]
Oct 30 13:50:28.864148 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:28.864083    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:50:28.864267 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:28.864198    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:50:28.864267 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:28.864242    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:50:29.769152 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:29.768288    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj" event={"ID":"f1b0539d-aead-4e37-b4ba-d6de98ae472b","Type":"ContainerStarted","Data":"bebfc86760169ac96e7eba6955608f51ee8a4b44910f4ad70af3ac0ae73b004c"}
Oct 30 13:50:29.769152 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:29.768370    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj" event={"ID":"f1b0539d-aead-4e37-b4ba-d6de98ae472b","Type":"ContainerStarted","Data":"6e6679b43f256b62b539fb8096db94503c65f60cfc536154764a39e09127b628"}
Oct 30 13:50:30.408238 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:30.408198    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj"
Oct 30 13:50:30.415848 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:30.415815    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj"
Oct 30 13:50:30.478456 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:30.478399    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj" podStartSLOduration=3.478380665 podStartE2EDuration="3.478380665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:50:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:50:29.844755342 +0000 UTC m=+2848.992022585" watchObservedRunningTime="2026-10-30 13:50:30.478380665 +0000 UTC m=+2849.625647905"
Oct 30 13:50:30.776280 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:30.776238    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj"
Oct 30 13:50:30.777355 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:30.777335    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj"
Oct 30 13:50:34.818144 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:34.818094    5093 generic.go:356] "Generic (PLEG): container finished" podID="4248f1a1-17bb-439c-80e5-8952de6299eb" containerID="fb026cb86cadcf995e83773f4ace93a70ae80e522d3f5db954cae066427f38cf" exitCode=0
Oct 30 13:50:34.818632 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:34.818180    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v" event={"ID":"4248f1a1-17bb-439c-80e5-8952de6299eb","Type":"ContainerDied","Data":"fb026cb86cadcf995e83773f4ace93a70ae80e522d3f5db954cae066427f38cf"}
Oct 30 13:50:34.818749 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:34.818725    5093 scope.go:122] "RemoveContainer" containerID="fb026cb86cadcf995e83773f4ace93a70ae80e522d3f5db954cae066427f38cf"
Oct 30 13:50:34.820897 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:34.820857    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:50:34.821023 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:34.820954    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:50:34.821023 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:34.821003    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:50:35.829667 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:35.829631    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v" event={"ID":"4248f1a1-17bb-439c-80e5-8952de6299eb","Type":"ContainerStarted","Data":"adcee79ecc2104e6d28b6da391753c52c13369052f24f674b818701afbb12864"}
Oct 30 13:50:36.119109 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:36.114584    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v"
Oct 30 13:50:36.119109 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:36.114715    5093 patch_prober.go:29] interesting pod/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 30 13:50:36.119109 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:36.114761    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v" podUID="4248f1a1-17bb-439c-80e5-8952de6299eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 30 13:50:37.120167 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:37.116538    5093 patch_prober.go:29] interesting pod/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 30 13:50:37.120167 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:37.116616    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v" podUID="4248f1a1-17bb-439c-80e5-8952de6299eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 30 13:50:38.172344 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:38.172272    5093 patch_prober.go:29] interesting pod/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 30 13:50:38.173057 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:38.173016    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v" podUID="4248f1a1-17bb-439c-80e5-8952de6299eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 30 13:50:39.112577 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:39.112539    5093 patch_prober.go:29] interesting pod/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 30 13:50:39.112801 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:39.112599    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v" podUID="4248f1a1-17bb-439c-80e5-8952de6299eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 30 13:50:40.112392 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:40.112356    5093 patch_prober.go:29] interesting pod/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 30 13:50:40.112824 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:40.112440    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v" podUID="4248f1a1-17bb-439c-80e5-8952de6299eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 30 13:50:41.112184 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:41.112146    5093 patch_prober.go:29] interesting pod/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 30 13:50:41.112370 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:41.112209    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v" podUID="4248f1a1-17bb-439c-80e5-8952de6299eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 30 13:50:42.138568 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:42.138522    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v"
Oct 30 13:50:42.139297 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:42.139263    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v"
Oct 30 13:50:42.140328 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:50:42.140301    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v"
Oct 30 13:50:51.169307 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:50:51.169265    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:51:04.652745 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:51:04.652706    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-996b8b-56c78549fc-t7vl6"]
Oct 30 13:51:04.659230 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:51:04.659203    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-996b8b-56c78549fc-t7vl6"
Oct 30 13:51:04.679729 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:51:04.679696    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-996b8b-serving-cert\""
Oct 30 13:51:04.682530 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:51:04.682497    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-996b8b-56c78549fc-t7vl6"]
Oct 30 13:51:04.761749 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:51:04.761706    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412-metrics-certs\") pod \"maas-api-e2e-inf-b-996b8b-56c78549fc-t7vl6\" (UID: \"dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-996b8b-56c78549fc-t7vl6"
Oct 30 13:51:04.761961 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:51:04.761825    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412-maas-api-tls\") pod \"maas-api-e2e-inf-b-996b8b-56c78549fc-t7vl6\" (UID: \"dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-996b8b-56c78549fc-t7vl6"
Oct 30 13:51:04.761961 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:51:04.761903    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412-openshift-service-ca\") pod \"maas-api-e2e-inf-b-996b8b-56c78549fc-t7vl6\" (UID: \"dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-996b8b-56c78549fc-t7vl6"
Oct 30 13:51:04.762151 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:51:04.762119    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j9nz\" (UniqueName: \"kubernetes.io/projected/dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412-kube-api-access-5j9nz\") pod \"maas-api-e2e-inf-b-996b8b-56c78549fc-t7vl6\" (UID: \"dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-996b8b-56c78549fc-t7vl6"
Oct 30 13:51:04.863284 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:51:04.863248    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412-maas-api-tls\") pod \"maas-api-e2e-inf-b-996b8b-56c78549fc-t7vl6\" (UID: \"dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-996b8b-56c78549fc-t7vl6"
Oct 30 13:51:04.863501 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:51:04.863334    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412-openshift-service-ca\") pod \"maas-api-e2e-inf-b-996b8b-56c78549fc-t7vl6\" (UID: \"dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-996b8b-56c78549fc-t7vl6"
Oct 30 13:51:04.863575 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:51:04.863504    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j9nz\" (UniqueName: \"kubernetes.io/projected/dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412-kube-api-access-5j9nz\") pod \"maas-api-e2e-inf-b-996b8b-56c78549fc-t7vl6\" (UID: \"dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-996b8b-56c78549fc-t7vl6"
Oct 30 13:51:04.863635 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:51:04.863605    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412-metrics-certs\") pod \"maas-api-e2e-inf-b-996b8b-56c78549fc-t7vl6\" (UID: \"dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-996b8b-56c78549fc-t7vl6"
Oct 30 13:51:04.864239 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:51:04.864202    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412-openshift-service-ca\") pod \"maas-api-e2e-inf-b-996b8b-56c78549fc-t7vl6\" (UID: \"dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-996b8b-56c78549fc-t7vl6"
Oct 30 13:51:04.866492 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:51:04.866462    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412-maas-api-tls\") pod \"maas-api-e2e-inf-b-996b8b-56c78549fc-t7vl6\" (UID: \"dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-996b8b-56c78549fc-t7vl6"
Oct 30 13:51:04.866665 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:51:04.866639    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412-metrics-certs\") pod \"maas-api-e2e-inf-b-996b8b-56c78549fc-t7vl6\" (UID: \"dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-996b8b-56c78549fc-t7vl6"
Oct 30 13:51:04.884030 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:51:04.883999    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j9nz\" (UniqueName: \"kubernetes.io/projected/dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412-kube-api-access-5j9nz\") pod \"maas-api-e2e-inf-b-996b8b-56c78549fc-t7vl6\" (UID: \"dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-996b8b-56c78549fc-t7vl6"
Oct 30 13:51:04.978294 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:51:04.977616    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-996b8b-56c78549fc-t7vl6"
Oct 30 13:51:05.294064 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:51:05.294026    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-996b8b-56c78549fc-t7vl6"]
Oct 30 13:51:05.294232 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:51:05.294178    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddead8a2f_bc4e_4ae2_aac0_bcb0dd28a412.slice/crio-4530d9cab6021e85ecd7edfdd9ccd3df7cbd5e1bfe7023c9545bda1184ef15e3 WatchSource:0}: Error finding container 4530d9cab6021e85ecd7edfdd9ccd3df7cbd5e1bfe7023c9545bda1184ef15e3: Status 404 returned error can't find the container with id 4530d9cab6021e85ecd7edfdd9ccd3df7cbd5e1bfe7023c9545bda1184ef15e3
Oct 30 13:51:05.295722 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:51:05.295703    5093 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 13:51:06.223957 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:51:06.223910    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-996b8b-56c78549fc-t7vl6" event={"ID":"dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412","Type":"ContainerStarted","Data":"682ee22f0f19cdad84b8eb4c2260f157f2096988799597c6e0242f45240d0456"}
Oct 30 13:51:06.224516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:51:06.224006    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-996b8b-56c78549fc-t7vl6"
Oct 30 13:51:06.224516 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:51:06.224023    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-996b8b-56c78549fc-t7vl6" event={"ID":"dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412","Type":"ContainerStarted","Data":"4530d9cab6021e85ecd7edfdd9ccd3df7cbd5e1bfe7023c9545bda1184ef15e3"}
Oct 30 13:51:06.271478 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:51:06.271415    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-996b8b-56c78549fc-t7vl6" podStartSLOduration=1.868937233 podStartE2EDuration="2.271399995s" totalImagesPullingTime="402.462762ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:51:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:51:06.268092408 +0000 UTC m=+2885.415359649" watchObservedRunningTime="2026-10-30 13:51:06.271399995 +0000 UTC m=+2885.418667236"
Oct 30 13:51:12.239660 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:51:12.239625    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-996b8b-56c78549fc-t7vl6"
Oct 30 13:51:21.170722 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:51:21.170675    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:51:51.172318 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:51:51.172224    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:52:07.566250 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:07.566205    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-926ff0-6b69d56f7f-cbkkk"]
Oct 30 13:52:07.694645 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:07.694606    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-926ff0-6b69d56f7f-cbkkk"]
Oct 30 13:52:07.706162 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:07.694974    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-926ff0-6b69d56f7f-cbkkk"
Oct 30 13:52:07.749154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:07.724486    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80042047-5b26-4633-98f6-d0ee98d70d11-openshift-service-ca\") pod \"maas-api-e2e-inf-a-926ff0-6b69d56f7f-cbkkk\" (UID: \"80042047-5b26-4633-98f6-d0ee98d70d11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-926ff0-6b69d56f7f-cbkkk"
Oct 30 13:52:07.803956 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:07.803918    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-926ff0-serving-cert\""
Oct 30 13:52:07.830214 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:07.824258    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80042047-5b26-4633-98f6-d0ee98d70d11-maas-api-tls\") pod \"maas-api-e2e-inf-a-926ff0-6b69d56f7f-cbkkk\" (UID: \"80042047-5b26-4633-98f6-d0ee98d70d11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-926ff0-6b69d56f7f-cbkkk"
Oct 30 13:52:07.857156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:07.856355    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zk44\" (UniqueName: \"kubernetes.io/projected/80042047-5b26-4633-98f6-d0ee98d70d11-kube-api-access-2zk44\") pod \"maas-api-e2e-inf-a-926ff0-6b69d56f7f-cbkkk\" (UID: \"80042047-5b26-4633-98f6-d0ee98d70d11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-926ff0-6b69d56f7f-cbkkk"
Oct 30 13:52:07.857156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:07.856518    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80042047-5b26-4633-98f6-d0ee98d70d11-metrics-certs\") pod \"maas-api-e2e-inf-a-926ff0-6b69d56f7f-cbkkk\" (UID: \"80042047-5b26-4633-98f6-d0ee98d70d11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-926ff0-6b69d56f7f-cbkkk"
Oct 30 13:52:07.975538 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:07.975493    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80042047-5b26-4633-98f6-d0ee98d70d11-openshift-service-ca\") pod \"maas-api-e2e-inf-a-926ff0-6b69d56f7f-cbkkk\" (UID: \"80042047-5b26-4633-98f6-d0ee98d70d11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-926ff0-6b69d56f7f-cbkkk"
Oct 30 13:52:07.975824 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:07.975802    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80042047-5b26-4633-98f6-d0ee98d70d11-maas-api-tls\") pod \"maas-api-e2e-inf-a-926ff0-6b69d56f7f-cbkkk\" (UID: \"80042047-5b26-4633-98f6-d0ee98d70d11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-926ff0-6b69d56f7f-cbkkk"
Oct 30 13:52:07.976106 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:07.976087    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zk44\" (UniqueName: \"kubernetes.io/projected/80042047-5b26-4633-98f6-d0ee98d70d11-kube-api-access-2zk44\") pod \"maas-api-e2e-inf-a-926ff0-6b69d56f7f-cbkkk\" (UID: \"80042047-5b26-4633-98f6-d0ee98d70d11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-926ff0-6b69d56f7f-cbkkk"
Oct 30 13:52:07.976341 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:07.976322    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80042047-5b26-4633-98f6-d0ee98d70d11-metrics-certs\") pod \"maas-api-e2e-inf-a-926ff0-6b69d56f7f-cbkkk\" (UID: \"80042047-5b26-4633-98f6-d0ee98d70d11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-926ff0-6b69d56f7f-cbkkk"
Oct 30 13:52:07.983815 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:07.983777    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80042047-5b26-4633-98f6-d0ee98d70d11-openshift-service-ca\") pod \"maas-api-e2e-inf-a-926ff0-6b69d56f7f-cbkkk\" (UID: \"80042047-5b26-4633-98f6-d0ee98d70d11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-926ff0-6b69d56f7f-cbkkk"
Oct 30 13:52:07.986419 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:07.986392    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80042047-5b26-4633-98f6-d0ee98d70d11-metrics-certs\") pod \"maas-api-e2e-inf-a-926ff0-6b69d56f7f-cbkkk\" (UID: \"80042047-5b26-4633-98f6-d0ee98d70d11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-926ff0-6b69d56f7f-cbkkk"
Oct 30 13:52:07.987244 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:07.987218    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80042047-5b26-4633-98f6-d0ee98d70d11-maas-api-tls\") pod \"maas-api-e2e-inf-a-926ff0-6b69d56f7f-cbkkk\" (UID: \"80042047-5b26-4633-98f6-d0ee98d70d11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-926ff0-6b69d56f7f-cbkkk"
Oct 30 13:52:08.006560 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:08.006532    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zk44\" (UniqueName: \"kubernetes.io/projected/80042047-5b26-4633-98f6-d0ee98d70d11-kube-api-access-2zk44\") pod \"maas-api-e2e-inf-a-926ff0-6b69d56f7f-cbkkk\" (UID: \"80042047-5b26-4633-98f6-d0ee98d70d11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-926ff0-6b69d56f7f-cbkkk"
Oct 30 13:52:08.222886 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:08.222845    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-926ff0-6b69d56f7f-cbkkk"
Oct 30 13:52:09.094154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:09.088672    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-926ff0-6b69d56f7f-cbkkk"]
Oct 30 13:52:10.164169 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:10.163848    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-926ff0-6b69d56f7f-cbkkk" event={"ID":"80042047-5b26-4633-98f6-d0ee98d70d11","Type":"ContainerStarted","Data":"2fd1128bba46f78007ecca3375dd90105b3aff8e94885e43648af5ff748e3a12"}
Oct 30 13:52:10.164169 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:10.163995    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-926ff0-6b69d56f7f-cbkkk" event={"ID":"80042047-5b26-4633-98f6-d0ee98d70d11","Type":"ContainerStarted","Data":"a90136063588e94f9675cde3aa6f8105cd717ac9e49ceb2c09a714e9b427493e"}
Oct 30 13:52:10.289156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:10.251304    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-926ff0-6b69d56f7f-cbkkk" podStartSLOduration=2.83114331 podStartE2EDuration="3.251284116s" totalImagesPullingTime="420.140806ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:52:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:52:10.24859869 +0000 UTC m=+2949.395865932" watchObservedRunningTime="2026-10-30 13:52:10.251284116 +0000 UTC m=+2949.398551357"
Oct 30 13:52:11.172806 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:11.172770    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-926ff0-6b69d56f7f-cbkkk"
Oct 30 13:52:16.825411 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:16.661768    5093 patch_prober.go:29] interesting pod/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": read tcp 10.132.0.2:52012->10.132.0.180:15021: read: connection reset by peer" start-of-body=
Oct 30 13:52:16.825411 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:16.661833    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj" podUID="f1b0539d-aead-4e37-b4ba-d6de98ae472b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": read tcp 10.132.0.2:52012->10.132.0.180:15021: read: connection reset by peer"
Oct 30 13:52:17.187650 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:17.187618    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-926ff0-6b69d56f7f-cbkkk"
Oct 30 13:52:17.239899 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:17.239861    5093 generic.go:356] "Generic (PLEG): container finished" podID="f1b0539d-aead-4e37-b4ba-d6de98ae472b" containerID="bebfc86760169ac96e7eba6955608f51ee8a4b44910f4ad70af3ac0ae73b004c" exitCode=0
Oct 30 13:52:17.240250 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:17.240100    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj" event={"ID":"f1b0539d-aead-4e37-b4ba-d6de98ae472b","Type":"ContainerDied","Data":"bebfc86760169ac96e7eba6955608f51ee8a4b44910f4ad70af3ac0ae73b004c"}
Oct 30 13:52:17.240691 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:17.240672    5093 scope.go:122] "RemoveContainer" containerID="bebfc86760169ac96e7eba6955608f51ee8a4b44910f4ad70af3ac0ae73b004c"
Oct 30 13:52:17.243098 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:17.243063    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:52:17.243250 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:17.243184    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:52:17.243250 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:17.243225    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:52:18.254757 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:18.254708    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj" event={"ID":"f1b0539d-aead-4e37-b4ba-d6de98ae472b","Type":"ContainerStarted","Data":"ee3f61d071e8821dca0897b446fc8248dd6d5716db4e988b4ce2727f5d28158e"}
Oct 30 13:52:18.407806 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:18.407767    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj"
Oct 30 13:52:18.408061 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:18.408033    5093 patch_prober.go:29] interesting pod/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 30 13:52:18.408123 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:18.408100    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj" podUID="f1b0539d-aead-4e37-b4ba-d6de98ae472b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 30 13:52:18.641158 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:18.637543    5093 patch_prober.go:29] interesting pod/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:52:18.641158 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:18.637618    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2" podUID="abe9a708-c8aa-414b-80a8-fcd5c1142425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:52:19.288176 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:19.287941    5093 generic.go:356] "Generic (PLEG): container finished" podID="abe9a708-c8aa-414b-80a8-fcd5c1142425" containerID="6813c626f4ef36b3ac1dedde6e29d5ae2cd0db66fed3061904338d971f29665f" exitCode=0
Oct 30 13:52:19.289020 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:19.288618    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2" event={"ID":"abe9a708-c8aa-414b-80a8-fcd5c1142425","Type":"ContainerDied","Data":"6813c626f4ef36b3ac1dedde6e29d5ae2cd0db66fed3061904338d971f29665f"}
Oct 30 13:52:19.289020 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:19.288986    5093 scope.go:122] "RemoveContainer" containerID="6813c626f4ef36b3ac1dedde6e29d5ae2cd0db66fed3061904338d971f29665f"
Oct 30 13:52:19.291145 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:19.291087    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:52:19.291290 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:19.291205    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:52:19.291290 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:19.291257    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:52:19.408718 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:19.408421    5093 patch_prober.go:29] interesting pod/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 30 13:52:19.408718 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:19.408484    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj" podUID="f1b0539d-aead-4e37-b4ba-d6de98ae472b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 30 13:52:20.299311 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:20.299271    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2" event={"ID":"abe9a708-c8aa-414b-80a8-fcd5c1142425","Type":"ContainerStarted","Data":"372e6576dcdda14a024d117f05d63750f8b5e944048018d23df303ef93771a6b"}
Oct 30 13:52:20.407762 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:20.407724    5093 patch_prober.go:29] interesting pod/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 30 13:52:20.407940 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:20.407780    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj" podUID="f1b0539d-aead-4e37-b4ba-d6de98ae472b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 30 13:52:20.557214 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:20.556718    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2"
Oct 30 13:52:20.557214 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:20.556816    5093 patch_prober.go:29] interesting pod/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 30 13:52:20.557214 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:20.556889    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2" podUID="abe9a708-c8aa-414b-80a8-fcd5c1142425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 30 13:52:21.215632 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:52:21.215595    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:52:21.407950 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:21.407903    5093 patch_prober.go:29] interesting pod/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 30 13:52:21.408416 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:21.407984    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj" podUID="f1b0539d-aead-4e37-b4ba-d6de98ae472b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 30 13:52:21.541218 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:21.539615    5093 patch_prober.go:29] interesting pod/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 30 13:52:21.541218 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:21.539703    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2" podUID="abe9a708-c8aa-414b-80a8-fcd5c1142425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 30 13:52:22.407602 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:22.407554    5093 patch_prober.go:29] interesting pod/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 30 13:52:22.407841 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:22.407628    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj" podUID="f1b0539d-aead-4e37-b4ba-d6de98ae472b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 30 13:52:22.546156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:22.540564    5093 patch_prober.go:29] interesting pod/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 30 13:52:22.546156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:22.540632    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2" podUID="abe9a708-c8aa-414b-80a8-fcd5c1142425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 30 13:52:23.407964 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:23.407925    5093 patch_prober.go:29] interesting pod/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 30 13:52:23.408285 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:23.407981    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj" podUID="f1b0539d-aead-4e37-b4ba-d6de98ae472b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 30 13:52:23.558157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:23.546553    5093 patch_prober.go:29] interesting pod/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 30 13:52:23.558157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:23.546646    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2" podUID="abe9a708-c8aa-414b-80a8-fcd5c1142425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 30 13:52:24.408245 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:24.408189    5093 patch_prober.go:29] interesting pod/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 30 13:52:24.408441 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:24.408285    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj" podUID="f1b0539d-aead-4e37-b4ba-d6de98ae472b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 30 13:52:24.546224 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:24.546124    5093 patch_prober.go:29] interesting pod/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 30 13:52:24.546552 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:24.546515    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2" podUID="abe9a708-c8aa-414b-80a8-fcd5c1142425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 30 13:52:25.495578 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:25.495532    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj"
Oct 30 13:52:25.496448 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:25.496421    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj"
Oct 30 13:52:25.498031 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:25.498012    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj"
Oct 30 13:52:25.542649 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:25.542607    5093 patch_prober.go:29] interesting pod/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 30 13:52:25.542964 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:25.542933    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2" podUID="abe9a708-c8aa-414b-80a8-fcd5c1142425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 30 13:52:26.540157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:26.539670    5093 patch_prober.go:29] interesting pod/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 30 13:52:26.540157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:26.539737    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2" podUID="abe9a708-c8aa-414b-80a8-fcd5c1142425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 30 13:52:27.546328 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:27.546288    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2"
Oct 30 13:52:27.546780 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:27.546760    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2"
Oct 30 13:52:27.547735 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:27.547710    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2"
Oct 30 13:52:31.003738 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:31.003705    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-996b8b/test-model-996b8b-kserve-b594c475-hrhvr"]
Oct 30 13:52:31.014502 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:31.014464    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-996b8b/test-model-996b8b-kserve-b594c475-hrhvr"
Oct 30 13:52:31.018571 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:31.018543    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-996b8b/test-model-996b8b-kserve-b594c475-hrhvr"]
Oct 30 13:52:31.030300 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:31.030270    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-996b8b\"/\"openshift-service-ca.crt\""
Oct 30 13:52:31.031122 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:31.030617    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-996b8b\"/\"kube-root-ca.crt\""
Oct 30 13:52:31.043587 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:31.043561    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-996b8b\"/\"test-model-996b8b-kserve-self-signed-certs\""
Oct 30 13:52:31.043732 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:31.043632    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-996b8b\"/\"default-dockercfg-l2hnf\""
Oct 30 13:52:31.087620 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:31.087566    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e45d640-397e-420f-b746-c25bc3e96739-home\") pod \"test-model-996b8b-kserve-b594c475-hrhvr\" (UID: \"6e45d640-397e-420f-b746-c25bc3e96739\") " pod="ai-tenant-e2e-inf-b-996b8b/test-model-996b8b-kserve-b594c475-hrhvr"
Oct 30 13:52:31.087937 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:31.087909    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e45d640-397e-420f-b746-c25bc3e96739-tmp-dir\") pod \"test-model-996b8b-kserve-b594c475-hrhvr\" (UID: \"6e45d640-397e-420f-b746-c25bc3e96739\") " pod="ai-tenant-e2e-inf-b-996b8b/test-model-996b8b-kserve-b594c475-hrhvr"
Oct 30 13:52:31.088101 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:31.088036    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e45d640-397e-420f-b746-c25bc3e96739-model-cache\") pod \"test-model-996b8b-kserve-b594c475-hrhvr\" (UID: \"6e45d640-397e-420f-b746-c25bc3e96739\") " pod="ai-tenant-e2e-inf-b-996b8b/test-model-996b8b-kserve-b594c475-hrhvr"
Oct 30 13:52:31.088101 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:31.088078    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e45d640-397e-420f-b746-c25bc3e96739-dshm\") pod \"test-model-996b8b-kserve-b594c475-hrhvr\" (UID: \"6e45d640-397e-420f-b746-c25bc3e96739\") " pod="ai-tenant-e2e-inf-b-996b8b/test-model-996b8b-kserve-b594c475-hrhvr"
Oct 30 13:52:31.088287 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:31.088240    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7s9c\" (UniqueName: \"kubernetes.io/projected/6e45d640-397e-420f-b746-c25bc3e96739-kube-api-access-d7s9c\") pod \"test-model-996b8b-kserve-b594c475-hrhvr\" (UID: \"6e45d640-397e-420f-b746-c25bc3e96739\") " pod="ai-tenant-e2e-inf-b-996b8b/test-model-996b8b-kserve-b594c475-hrhvr"
Oct 30 13:52:31.088366 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:31.088326    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e45d640-397e-420f-b746-c25bc3e96739-tls-certs\") pod \"test-model-996b8b-kserve-b594c475-hrhvr\" (UID: \"6e45d640-397e-420f-b746-c25bc3e96739\") " pod="ai-tenant-e2e-inf-b-996b8b/test-model-996b8b-kserve-b594c475-hrhvr"
Oct 30 13:52:31.189619 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:31.189583    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e45d640-397e-420f-b746-c25bc3e96739-model-cache\") pod \"test-model-996b8b-kserve-b594c475-hrhvr\" (UID: \"6e45d640-397e-420f-b746-c25bc3e96739\") " pod="ai-tenant-e2e-inf-b-996b8b/test-model-996b8b-kserve-b594c475-hrhvr"
Oct 30 13:52:31.189828 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:31.189629    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e45d640-397e-420f-b746-c25bc3e96739-dshm\") pod \"test-model-996b8b-kserve-b594c475-hrhvr\" (UID: \"6e45d640-397e-420f-b746-c25bc3e96739\") " pod="ai-tenant-e2e-inf-b-996b8b/test-model-996b8b-kserve-b594c475-hrhvr"
Oct 30 13:52:31.189828 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:31.189698    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7s9c\" (UniqueName: \"kubernetes.io/projected/6e45d640-397e-420f-b746-c25bc3e96739-kube-api-access-d7s9c\") pod \"test-model-996b8b-kserve-b594c475-hrhvr\" (UID: \"6e45d640-397e-420f-b746-c25bc3e96739\") " pod="ai-tenant-e2e-inf-b-996b8b/test-model-996b8b-kserve-b594c475-hrhvr"
Oct 30 13:52:31.189828 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:31.189745    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e45d640-397e-420f-b746-c25bc3e96739-tls-certs\") pod \"test-model-996b8b-kserve-b594c475-hrhvr\" (UID: \"6e45d640-397e-420f-b746-c25bc3e96739\") " pod="ai-tenant-e2e-inf-b-996b8b/test-model-996b8b-kserve-b594c475-hrhvr"
Oct 30 13:52:31.190007 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:31.189912    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e45d640-397e-420f-b746-c25bc3e96739-home\") pod \"test-model-996b8b-kserve-b594c475-hrhvr\" (UID: \"6e45d640-397e-420f-b746-c25bc3e96739\") " pod="ai-tenant-e2e-inf-b-996b8b/test-model-996b8b-kserve-b594c475-hrhvr"
Oct 30 13:52:31.190067 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:31.190023    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e45d640-397e-420f-b746-c25bc3e96739-model-cache\") pod \"test-model-996b8b-kserve-b594c475-hrhvr\" (UID: \"6e45d640-397e-420f-b746-c25bc3e96739\") " pod="ai-tenant-e2e-inf-b-996b8b/test-model-996b8b-kserve-b594c475-hrhvr"
Oct 30 13:52:31.190213 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:31.190170    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e45d640-397e-420f-b746-c25bc3e96739-tmp-dir\") pod \"test-model-996b8b-kserve-b594c475-hrhvr\" (UID: \"6e45d640-397e-420f-b746-c25bc3e96739\") " pod="ai-tenant-e2e-inf-b-996b8b/test-model-996b8b-kserve-b594c475-hrhvr"
Oct 30 13:52:31.190301 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:31.190239    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e45d640-397e-420f-b746-c25bc3e96739-home\") pod \"test-model-996b8b-kserve-b594c475-hrhvr\" (UID: \"6e45d640-397e-420f-b746-c25bc3e96739\") " pod="ai-tenant-e2e-inf-b-996b8b/test-model-996b8b-kserve-b594c475-hrhvr"
Oct 30 13:52:31.190426 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:31.190402    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e45d640-397e-420f-b746-c25bc3e96739-tmp-dir\") pod \"test-model-996b8b-kserve-b594c475-hrhvr\" (UID: \"6e45d640-397e-420f-b746-c25bc3e96739\") " pod="ai-tenant-e2e-inf-b-996b8b/test-model-996b8b-kserve-b594c475-hrhvr"
Oct 30 13:52:31.192056 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:31.192037    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e45d640-397e-420f-b746-c25bc3e96739-dshm\") pod \"test-model-996b8b-kserve-b594c475-hrhvr\" (UID: \"6e45d640-397e-420f-b746-c25bc3e96739\") " pod="ai-tenant-e2e-inf-b-996b8b/test-model-996b8b-kserve-b594c475-hrhvr"
Oct 30 13:52:31.192349 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:31.192331    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e45d640-397e-420f-b746-c25bc3e96739-tls-certs\") pod \"test-model-996b8b-kserve-b594c475-hrhvr\" (UID: \"6e45d640-397e-420f-b746-c25bc3e96739\") " pod="ai-tenant-e2e-inf-b-996b8b/test-model-996b8b-kserve-b594c475-hrhvr"
Oct 30 13:52:31.210605 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:31.210579    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7s9c\" (UniqueName: \"kubernetes.io/projected/6e45d640-397e-420f-b746-c25bc3e96739-kube-api-access-d7s9c\") pod \"test-model-996b8b-kserve-b594c475-hrhvr\" (UID: \"6e45d640-397e-420f-b746-c25bc3e96739\") " pod="ai-tenant-e2e-inf-b-996b8b/test-model-996b8b-kserve-b594c475-hrhvr"
Oct 30 13:52:31.334117 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:31.334021    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-996b8b/test-model-996b8b-kserve-b594c475-hrhvr"
Oct 30 13:52:31.666092 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:52:31.666058    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e45d640_397e_420f_b746_c25bc3e96739.slice/crio-5f8636809a04f4715fbffe38ba064ecc452f9cc2c3e50d379ba7169d1636a240 WatchSource:0}: Error finding container 5f8636809a04f4715fbffe38ba064ecc452f9cc2c3e50d379ba7169d1636a240: Status 404 returned error can't find the container with id 5f8636809a04f4715fbffe38ba064ecc452f9cc2c3e50d379ba7169d1636a240
Oct 30 13:52:31.668025 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:31.667995    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-996b8b/test-model-996b8b-kserve-b594c475-hrhvr"]
Oct 30 13:52:32.464622 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:32.464580    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-996b8b/test-model-996b8b-kserve-b594c475-hrhvr" event={"ID":"6e45d640-397e-420f-b746-c25bc3e96739","Type":"ContainerStarted","Data":"0ad255d3f2e25b7c91b6503d695904c383e90412917bca23500be2dee8d4b68c"}
Oct 30 13:52:32.465081 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:32.464633    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-996b8b/test-model-996b8b-kserve-b594c475-hrhvr" event={"ID":"6e45d640-397e-420f-b746-c25bc3e96739","Type":"ContainerStarted","Data":"5f8636809a04f4715fbffe38ba064ecc452f9cc2c3e50d379ba7169d1636a240"}
Oct 30 13:52:32.512035 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:32.511973    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-996b8b/test-model-996b8b-kserve-b594c475-hrhvr" podStartSLOduration=2.511958675 podStartE2EDuration="2.511958675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:52:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:52:32.507201258 +0000 UTC m=+2971.654468500" watchObservedRunningTime="2026-10-30 13:52:32.511958675 +0000 UTC m=+2971.659225912"
Oct 30 13:52:33.334492 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:33.334436    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-996b8b/test-model-996b8b-kserve-b594c475-hrhvr"
Oct 30 13:52:35.501012 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:35.500969    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-51605e/test-model-51605e-kserve-7d4cf4d78c-5527f"]
Oct 30 13:52:35.507317 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:35.507289    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-51605e/test-model-51605e-kserve-7d4cf4d78c-5527f"
Oct 30 13:52:35.528511 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:35.528476    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-51605e/test-model-51605e-kserve-7d4cf4d78c-5527f"]
Oct 30 13:52:35.533803 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:35.533770    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-51605e\"/\"kube-root-ca.crt\""
Oct 30 13:52:35.533803 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:35.533769    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-51605e\"/\"openshift-service-ca.crt\""
Oct 30 13:52:35.536208 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:35.536184    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-51605e\"/\"test-model-51605e-kserve-self-signed-certs\""
Oct 30 13:52:35.536318 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:35.536216    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-51605e\"/\"default-dockercfg-bpbhp\""
Oct 30 13:52:35.600078 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:35.600038    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5-tmp-dir\") pod \"test-model-51605e-kserve-7d4cf4d78c-5527f\" (UID: \"14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5\") " pod="ai-tenant-e2e-inf-a-51605e/test-model-51605e-kserve-7d4cf4d78c-5527f"
Oct 30 13:52:35.600303 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:35.600091    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w69cx\" (UniqueName: \"kubernetes.io/projected/14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5-kube-api-access-w69cx\") pod \"test-model-51605e-kserve-7d4cf4d78c-5527f\" (UID: \"14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5\") " pod="ai-tenant-e2e-inf-a-51605e/test-model-51605e-kserve-7d4cf4d78c-5527f"
Oct 30 13:52:35.600303 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:35.600226    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5-dshm\") pod \"test-model-51605e-kserve-7d4cf4d78c-5527f\" (UID: \"14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5\") " pod="ai-tenant-e2e-inf-a-51605e/test-model-51605e-kserve-7d4cf4d78c-5527f"
Oct 30 13:52:35.600303 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:35.600280    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5-tls-certs\") pod \"test-model-51605e-kserve-7d4cf4d78c-5527f\" (UID: \"14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5\") " pod="ai-tenant-e2e-inf-a-51605e/test-model-51605e-kserve-7d4cf4d78c-5527f"
Oct 30 13:52:35.600442 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:35.600383    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5-model-cache\") pod \"test-model-51605e-kserve-7d4cf4d78c-5527f\" (UID: \"14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5\") " pod="ai-tenant-e2e-inf-a-51605e/test-model-51605e-kserve-7d4cf4d78c-5527f"
Oct 30 13:52:35.600442 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:35.600427    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5-home\") pod \"test-model-51605e-kserve-7d4cf4d78c-5527f\" (UID: \"14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5\") " pod="ai-tenant-e2e-inf-a-51605e/test-model-51605e-kserve-7d4cf4d78c-5527f"
Oct 30 13:52:35.702623 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:35.702578    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5-tmp-dir\") pod \"test-model-51605e-kserve-7d4cf4d78c-5527f\" (UID: \"14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5\") " pod="ai-tenant-e2e-inf-a-51605e/test-model-51605e-kserve-7d4cf4d78c-5527f"
Oct 30 13:52:35.702846 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:35.702644    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w69cx\" (UniqueName: \"kubernetes.io/projected/14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5-kube-api-access-w69cx\") pod \"test-model-51605e-kserve-7d4cf4d78c-5527f\" (UID: \"14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5\") " pod="ai-tenant-e2e-inf-a-51605e/test-model-51605e-kserve-7d4cf4d78c-5527f"
Oct 30 13:52:35.702846 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:35.702694    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5-dshm\") pod \"test-model-51605e-kserve-7d4cf4d78c-5527f\" (UID: \"14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5\") " pod="ai-tenant-e2e-inf-a-51605e/test-model-51605e-kserve-7d4cf4d78c-5527f"
Oct 30 13:52:35.702846 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:35.702734    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5-tls-certs\") pod \"test-model-51605e-kserve-7d4cf4d78c-5527f\" (UID: \"14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5\") " pod="ai-tenant-e2e-inf-a-51605e/test-model-51605e-kserve-7d4cf4d78c-5527f"
Oct 30 13:52:35.702846 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:35.702772    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5-model-cache\") pod \"test-model-51605e-kserve-7d4cf4d78c-5527f\" (UID: \"14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5\") " pod="ai-tenant-e2e-inf-a-51605e/test-model-51605e-kserve-7d4cf4d78c-5527f"
Oct 30 13:52:35.702846 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:35.702801    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5-home\") pod \"test-model-51605e-kserve-7d4cf4d78c-5527f\" (UID: \"14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5\") " pod="ai-tenant-e2e-inf-a-51605e/test-model-51605e-kserve-7d4cf4d78c-5527f"
Oct 30 13:52:35.703276 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:35.703244    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5-home\") pod \"test-model-51605e-kserve-7d4cf4d78c-5527f\" (UID: \"14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5\") " pod="ai-tenant-e2e-inf-a-51605e/test-model-51605e-kserve-7d4cf4d78c-5527f"
Oct 30 13:52:35.703562 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:35.703537    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5-tmp-dir\") pod \"test-model-51605e-kserve-7d4cf4d78c-5527f\" (UID: \"14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5\") " pod="ai-tenant-e2e-inf-a-51605e/test-model-51605e-kserve-7d4cf4d78c-5527f"
Oct 30 13:52:35.703681 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:35.703659    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5-model-cache\") pod \"test-model-51605e-kserve-7d4cf4d78c-5527f\" (UID: \"14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5\") " pod="ai-tenant-e2e-inf-a-51605e/test-model-51605e-kserve-7d4cf4d78c-5527f"
Oct 30 13:52:35.705518 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:35.705488    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5-dshm\") pod \"test-model-51605e-kserve-7d4cf4d78c-5527f\" (UID: \"14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5\") " pod="ai-tenant-e2e-inf-a-51605e/test-model-51605e-kserve-7d4cf4d78c-5527f"
Oct 30 13:52:35.705921 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:35.705898    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5-tls-certs\") pod \"test-model-51605e-kserve-7d4cf4d78c-5527f\" (UID: \"14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5\") " pod="ai-tenant-e2e-inf-a-51605e/test-model-51605e-kserve-7d4cf4d78c-5527f"
Oct 30 13:52:35.777391 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:35.777306    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w69cx\" (UniqueName: \"kubernetes.io/projected/14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5-kube-api-access-w69cx\") pod \"test-model-51605e-kserve-7d4cf4d78c-5527f\" (UID: \"14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5\") " pod="ai-tenant-e2e-inf-a-51605e/test-model-51605e-kserve-7d4cf4d78c-5527f"
Oct 30 13:52:35.835614 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:35.835575    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-51605e/test-model-51605e-kserve-7d4cf4d78c-5527f"
Oct 30 13:52:36.152554 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:36.152524    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-51605e/test-model-51605e-kserve-7d4cf4d78c-5527f"]
Oct 30 13:52:36.153062 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:52:36.153036    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14eb7d03_231f_46f2_aa9c_9f00a1e9a3b5.slice/crio-fdf4a7c05f86010b7dc6c3a98dba91874109785de72c766ee4e17be7ae382382 WatchSource:0}: Error finding container fdf4a7c05f86010b7dc6c3a98dba91874109785de72c766ee4e17be7ae382382: Status 404 returned error can't find the container with id fdf4a7c05f86010b7dc6c3a98dba91874109785de72c766ee4e17be7ae382382
Oct 30 13:52:36.622723 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:36.622684    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-51605e/test-model-51605e-kserve-7d4cf4d78c-5527f" event={"ID":"14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5","Type":"ContainerStarted","Data":"09963cb8c2e410658cbdd0cb2f4663702174a43f3370a92cbbd37a915bd87e23"}
Oct 30 13:52:36.623112 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:36.622734    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-51605e/test-model-51605e-kserve-7d4cf4d78c-5527f" event={"ID":"14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5","Type":"ContainerStarted","Data":"fdf4a7c05f86010b7dc6c3a98dba91874109785de72c766ee4e17be7ae382382"}
Oct 30 13:52:36.668670 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:36.668600    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-51605e/test-model-51605e-kserve-7d4cf4d78c-5527f" podStartSLOduration=1.6685860620000001 podStartE2EDuration="1.668586062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:52:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:52:36.664434641 +0000 UTC m=+2975.811701883" watchObservedRunningTime="2026-10-30 13:52:36.668586062 +0000 UTC m=+2975.815853303"
Oct 30 13:52:36.836032 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:36.835996    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-51605e/test-model-51605e-kserve-7d4cf4d78c-5527f"
Oct 30 13:52:41.334582 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:41.334505    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-996b8b/test-model-996b8b-kserve-b594c475-hrhvr"
Oct 30 13:52:41.348871 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:41.348842    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-996b8b/test-model-996b8b-kserve-b594c475-hrhvr"
Oct 30 13:52:41.685292 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:41.685254    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-996b8b/test-model-996b8b-kserve-b594c475-hrhvr"
Oct 30 13:52:45.836154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:45.836090    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-51605e/test-model-51605e-kserve-7d4cf4d78c-5527f"
Oct 30 13:52:45.852248 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:45.852202    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-51605e/test-model-51605e-kserve-7d4cf4d78c-5527f"
Oct 30 13:52:46.733442 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:52:46.733409    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-51605e/test-model-51605e-kserve-7d4cf4d78c-5527f"
Oct 30 13:52:51.169602 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:52:51.169556    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:53:01.661431 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:53:01.661399    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jdj4_f93ee017-ab79-447c-90e0-5b85b71dba24/console-operator/1.log"
Oct 30 13:53:01.667789 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:53:01.667747    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tnm_3a675ac7-fbe1-449b-accf-bb0456ed0bd3/ovn-acl-logging/0.log"
Oct 30 13:53:01.668706 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:53:01.668681    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jdj4_f93ee017-ab79-447c-90e0-5b85b71dba24/console-operator/1.log"
Oct 30 13:53:01.682453 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:53:01.682423    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tnm_3a675ac7-fbe1-449b-accf-bb0456ed0bd3/ovn-acl-logging/0.log"
Oct 30 13:53:14.406454 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:53:14.406234    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-09014b-7d8bb679ff-msw5h"]
Oct 30 13:53:14.414704 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:53:14.414673    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09014b-7d8bb679ff-msw5h"
Oct 30 13:53:14.419492 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:53:14.419443    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-09014b-7d8bb679ff-msw5h"]
Oct 30 13:53:14.453170 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:53:14.443171    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-09014b-serving-cert\""
Oct 30 13:53:14.598185 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:53:14.581402    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19dc1706-24ed-4126-8ee7-b35fd6fe9d37-maas-api-tls\") pod \"maas-api-e2e-inf-b-09014b-7d8bb679ff-msw5h\" (UID: \"19dc1706-24ed-4126-8ee7-b35fd6fe9d37\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09014b-7d8bb679ff-msw5h"
Oct 30 13:53:14.598185 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:53:14.581603    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8r6x\" (UniqueName: \"kubernetes.io/projected/19dc1706-24ed-4126-8ee7-b35fd6fe9d37-kube-api-access-g8r6x\") pod \"maas-api-e2e-inf-b-09014b-7d8bb679ff-msw5h\" (UID: \"19dc1706-24ed-4126-8ee7-b35fd6fe9d37\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09014b-7d8bb679ff-msw5h"
Oct 30 13:53:14.598185 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:53:14.581772    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19dc1706-24ed-4126-8ee7-b35fd6fe9d37-openshift-service-ca\") pod \"maas-api-e2e-inf-b-09014b-7d8bb679ff-msw5h\" (UID: \"19dc1706-24ed-4126-8ee7-b35fd6fe9d37\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09014b-7d8bb679ff-msw5h"
Oct 30 13:53:14.598185 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:53:14.581820    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19dc1706-24ed-4126-8ee7-b35fd6fe9d37-metrics-certs\") pod \"maas-api-e2e-inf-b-09014b-7d8bb679ff-msw5h\" (UID: \"19dc1706-24ed-4126-8ee7-b35fd6fe9d37\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09014b-7d8bb679ff-msw5h"
Oct 30 13:53:14.736156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:53:14.728928    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19dc1706-24ed-4126-8ee7-b35fd6fe9d37-openshift-service-ca\") pod \"maas-api-e2e-inf-b-09014b-7d8bb679ff-msw5h\" (UID: \"19dc1706-24ed-4126-8ee7-b35fd6fe9d37\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09014b-7d8bb679ff-msw5h"
Oct 30 13:53:14.736462 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:53:14.736430    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19dc1706-24ed-4126-8ee7-b35fd6fe9d37-metrics-certs\") pod \"maas-api-e2e-inf-b-09014b-7d8bb679ff-msw5h\" (UID: \"19dc1706-24ed-4126-8ee7-b35fd6fe9d37\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09014b-7d8bb679ff-msw5h"
Oct 30 13:53:14.736874 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:53:14.736845    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19dc1706-24ed-4126-8ee7-b35fd6fe9d37-maas-api-tls\") pod \"maas-api-e2e-inf-b-09014b-7d8bb679ff-msw5h\" (UID: \"19dc1706-24ed-4126-8ee7-b35fd6fe9d37\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09014b-7d8bb679ff-msw5h"
Oct 30 13:53:14.737177 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:53:14.737144    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8r6x\" (UniqueName: \"kubernetes.io/projected/19dc1706-24ed-4126-8ee7-b35fd6fe9d37-kube-api-access-g8r6x\") pod \"maas-api-e2e-inf-b-09014b-7d8bb679ff-msw5h\" (UID: \"19dc1706-24ed-4126-8ee7-b35fd6fe9d37\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09014b-7d8bb679ff-msw5h"
Oct 30 13:53:14.737709 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:53:14.736337    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19dc1706-24ed-4126-8ee7-b35fd6fe9d37-openshift-service-ca\") pod \"maas-api-e2e-inf-b-09014b-7d8bb679ff-msw5h\" (UID: \"19dc1706-24ed-4126-8ee7-b35fd6fe9d37\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09014b-7d8bb679ff-msw5h"
Oct 30 13:53:14.832154 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:53:14.831972    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19dc1706-24ed-4126-8ee7-b35fd6fe9d37-maas-api-tls\") pod \"maas-api-e2e-inf-b-09014b-7d8bb679ff-msw5h\" (UID: \"19dc1706-24ed-4126-8ee7-b35fd6fe9d37\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09014b-7d8bb679ff-msw5h"
Oct 30 13:53:14.855600 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:53:14.855562    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8r6x\" (UniqueName: \"kubernetes.io/projected/19dc1706-24ed-4126-8ee7-b35fd6fe9d37-kube-api-access-g8r6x\") pod \"maas-api-e2e-inf-b-09014b-7d8bb679ff-msw5h\" (UID: \"19dc1706-24ed-4126-8ee7-b35fd6fe9d37\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09014b-7d8bb679ff-msw5h"
Oct 30 13:53:14.865146 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:53:14.865036    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19dc1706-24ed-4126-8ee7-b35fd6fe9d37-metrics-certs\") pod \"maas-api-e2e-inf-b-09014b-7d8bb679ff-msw5h\" (UID: \"19dc1706-24ed-4126-8ee7-b35fd6fe9d37\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09014b-7d8bb679ff-msw5h"
Oct 30 13:53:15.114039 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:53:15.113940    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09014b-7d8bb679ff-msw5h"
Oct 30 13:53:15.489001 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:53:15.488175    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-09014b-7d8bb679ff-msw5h"]
Oct 30 13:53:16.049118 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:53:16.049076    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09014b-7d8bb679ff-msw5h" event={"ID":"19dc1706-24ed-4126-8ee7-b35fd6fe9d37","Type":"ContainerStarted","Data":"6a33f177bfefb51f7a3db270b3b5511a1988c5e6dab6485cb8860b5aa93ae54d"}
Oct 30 13:53:17.075173 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:53:17.071930    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09014b-7d8bb679ff-msw5h" event={"ID":"19dc1706-24ed-4126-8ee7-b35fd6fe9d37","Type":"ContainerStarted","Data":"5b0d9611fcf4ba7797041366cbd099031221f2c2b63cd4a1009acc896df1f0e0"}
Oct 30 13:53:17.075173 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:53:17.072402    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09014b-7d8bb679ff-msw5h"
Oct 30 13:53:17.127656 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:53:17.127593    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09014b-7d8bb679ff-msw5h" podStartSLOduration=2.678734074 podStartE2EDuration="3.127579143s" totalImagesPullingTime="448.845069ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:53:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:53:17.123477956 +0000 UTC m=+3016.270745199" watchObservedRunningTime="2026-10-30 13:53:17.127579143 +0000 UTC m=+3016.274846383"
Oct 30 13:53:21.176359 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:53:21.176319    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:53:23.089706 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:53:23.089673    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09014b-7d8bb679ff-msw5h"
Oct 30 13:53:41.206574 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:53:41.206533    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-d4ef2e/e2e-usage-sim-f3a8c8-8576b7fd6b-kdhdz"]
Oct 30 13:53:41.207028 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:53:41.206803    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-extmod-shr-user-d4ef2e/e2e-usage-sim-f3a8c8-8576b7fd6b-kdhdz" podUID="73976ccb-1e12-40ef-bf4b-6c5deeda3b88" containerName="sim" containerID="cri-o://3d46ad85b1a67a5010c1ad0f57cf70899bcee151fa7e817abfc26f0f18c2a26b" gracePeriod=30
Oct 30 13:53:41.707196 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:53:41.703976    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-d4ef2e/e2e-usage-sim-f3a8c8-8576b7fd6b-kdhdz"
Oct 30 13:53:41.727540 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:53:41.727113    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73976ccb-1e12-40ef-bf4b-6c5deeda3b88-tls\" (UniqueName: \"kubernetes.io/secret/73976ccb-1e12-40ef-bf4b-6c5deeda3b88-tls\") pod \"73976ccb-1e12-40ef-bf4b-6c5deeda3b88\" (UID: \"73976ccb-1e12-40ef-bf4b-6c5deeda3b88\") "
Oct 30 13:53:41.727878 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:53:41.727858    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73976ccb-1e12-40ef-bf4b-6c5deeda3b88-kube-api-access-bmsvt\" (UniqueName: \"kubernetes.io/projected/73976ccb-1e12-40ef-bf4b-6c5deeda3b88-kube-api-access-bmsvt\") pod \"73976ccb-1e12-40ef-bf4b-6c5deeda3b88\" (UID: \"73976ccb-1e12-40ef-bf4b-6c5deeda3b88\") "
Oct 30 13:53:41.731786 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:53:41.731743    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73976ccb-1e12-40ef-bf4b-6c5deeda3b88-kube-api-access-bmsvt" pod "73976ccb-1e12-40ef-bf4b-6c5deeda3b88" (UID: "73976ccb-1e12-40ef-bf4b-6c5deeda3b88"). InnerVolumeSpecName "kube-api-access-bmsvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:53:41.748478 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:53:41.745349    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73976ccb-1e12-40ef-bf4b-6c5deeda3b88-tls" pod "73976ccb-1e12-40ef-bf4b-6c5deeda3b88" (UID: "73976ccb-1e12-40ef-bf4b-6c5deeda3b88"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:53:41.830366 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:53:41.830265    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmsvt\" (UniqueName: \"kubernetes.io/projected/73976ccb-1e12-40ef-bf4b-6c5deeda3b88-kube-api-access-bmsvt\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:53:41.830366 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:53:41.830301    5093 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/73976ccb-1e12-40ef-bf4b-6c5deeda3b88-tls\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:53:42.307287 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:53:42.307238    5093 generic.go:356] "Generic (PLEG): container finished" podID="73976ccb-1e12-40ef-bf4b-6c5deeda3b88" containerID="3d46ad85b1a67a5010c1ad0f57cf70899bcee151fa7e817abfc26f0f18c2a26b" exitCode=0
Oct 30 13:53:42.307287 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:53:42.307287    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-d4ef2e/e2e-usage-sim-f3a8c8-8576b7fd6b-kdhdz"
Oct 30 13:53:42.307795 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:53:42.307312    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-d4ef2e/e2e-usage-sim-f3a8c8-8576b7fd6b-kdhdz" event={"ID":"73976ccb-1e12-40ef-bf4b-6c5deeda3b88","Type":"ContainerDied","Data":"3d46ad85b1a67a5010c1ad0f57cf70899bcee151fa7e817abfc26f0f18c2a26b"}
Oct 30 13:53:42.307795 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:53:42.307373    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-d4ef2e/e2e-usage-sim-f3a8c8-8576b7fd6b-kdhdz" event={"ID":"73976ccb-1e12-40ef-bf4b-6c5deeda3b88","Type":"ContainerDied","Data":"183c935559916ea624f6f807f35ce1d34ecbc0432cbd0fa2ef671db0a77e7704"}
Oct 30 13:53:42.307795 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:53:42.307397    5093 scope.go:122] "RemoveContainer" containerID="3d46ad85b1a67a5010c1ad0f57cf70899bcee151fa7e817abfc26f0f18c2a26b"
Oct 30 13:53:42.320441 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:53:42.320416    5093 scope.go:122] "RemoveContainer" containerID="3d46ad85b1a67a5010c1ad0f57cf70899bcee151fa7e817abfc26f0f18c2a26b"
Oct 30 13:53:42.320746 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:53:42.320724    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d46ad85b1a67a5010c1ad0f57cf70899bcee151fa7e817abfc26f0f18c2a26b\": container with ID starting with 3d46ad85b1a67a5010c1ad0f57cf70899bcee151fa7e817abfc26f0f18c2a26b not found: ID does not exist" containerID="3d46ad85b1a67a5010c1ad0f57cf70899bcee151fa7e817abfc26f0f18c2a26b"
Oct 30 13:53:42.320803 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:53:42.320755    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d46ad85b1a67a5010c1ad0f57cf70899bcee151fa7e817abfc26f0f18c2a26b"} err="failed to get container status \"3d46ad85b1a67a5010c1ad0f57cf70899bcee151fa7e817abfc26f0f18c2a26b\": rpc error: code = NotFound desc = could not find container \"3d46ad85b1a67a5010c1ad0f57cf70899bcee151fa7e817abfc26f0f18c2a26b\": container with ID starting with 3d46ad85b1a67a5010c1ad0f57cf70899bcee151fa7e817abfc26f0f18c2a26b not found: ID does not exist"
Oct 30 13:53:42.364988 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:53:42.364950    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-d4ef2e/e2e-usage-sim-f3a8c8-8576b7fd6b-kdhdz"]
Oct 30 13:53:42.377667 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:53:42.377634    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-extmod-shr-user-d4ef2e/e2e-usage-sim-f3a8c8-8576b7fd6b-kdhdz"]
Oct 30 13:53:43.001676 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:53:43.001641    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73976ccb-1e12-40ef-bf4b-6c5deeda3b88" path="/var/lib/kubelet/pods/73976ccb-1e12-40ef-bf4b-6c5deeda3b88/volumes"
Oct 30 13:53:51.201619 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:53:51.201573    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:54:01.889753 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:01.889709    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8"]
Oct 30 13:54:01.890449 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:01.890014    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8" podUID="428b29c0-c277-4e39-ab76-10655b7382ba" containerName="istio-proxy" containerID="cri-o://105319a27172cc8133bb2aaee44b096dc497b2ea3db368c2373fb0939178a819" gracePeriod=30
Oct 30 13:54:06.665744 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:06.665694    5093 patch_prober.go:29] interesting pod/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:54:06.666179 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:06.665773    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8" podUID="428b29c0-c277-4e39-ab76-10655b7382ba" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:54:07.240547 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:07.240511    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8"
Oct 30 13:54:07.285182 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:07.285114    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/428b29c0-c277-4e39-ab76-10655b7382ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/428b29c0-c277-4e39-ab76-10655b7382ba-istio-data\") pod \"428b29c0-c277-4e39-ab76-10655b7382ba\" (UID: \"428b29c0-c277-4e39-ab76-10655b7382ba\") "
Oct 30 13:54:07.285430 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:07.285218    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/428b29c0-c277-4e39-ab76-10655b7382ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/428b29c0-c277-4e39-ab76-10655b7382ba-workload-socket\") pod \"428b29c0-c277-4e39-ab76-10655b7382ba\" (UID: \"428b29c0-c277-4e39-ab76-10655b7382ba\") "
Oct 30 13:54:07.285430 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:07.285335    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/428b29c0-c277-4e39-ab76-10655b7382ba-kube-api-access-vmmrl\" (UniqueName: \"kubernetes.io/projected/428b29c0-c277-4e39-ab76-10655b7382ba-kube-api-access-vmmrl\") pod \"428b29c0-c277-4e39-ab76-10655b7382ba\" (UID: \"428b29c0-c277-4e39-ab76-10655b7382ba\") "
Oct 30 13:54:07.285430 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:07.285371    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/428b29c0-c277-4e39-ab76-10655b7382ba-istio-data" pod "428b29c0-c277-4e39-ab76-10655b7382ba" (UID: "428b29c0-c277-4e39-ab76-10655b7382ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:07.285430 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:07.285378    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/428b29c0-c277-4e39-ab76-10655b7382ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/428b29c0-c277-4e39-ab76-10655b7382ba-istio-envoy\") pod \"428b29c0-c277-4e39-ab76-10655b7382ba\" (UID: \"428b29c0-c277-4e39-ab76-10655b7382ba\") "
Oct 30 13:54:07.285573 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:07.285481    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/428b29c0-c277-4e39-ab76-10655b7382ba-workload-socket" pod "428b29c0-c277-4e39-ab76-10655b7382ba" (UID: "428b29c0-c277-4e39-ab76-10655b7382ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:07.285573 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:07.285521    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/428b29c0-c277-4e39-ab76-10655b7382ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/428b29c0-c277-4e39-ab76-10655b7382ba-credential-socket\") pod \"428b29c0-c277-4e39-ab76-10655b7382ba\" (UID: \"428b29c0-c277-4e39-ab76-10655b7382ba\") "
Oct 30 13:54:07.285646 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:07.285582    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/428b29c0-c277-4e39-ab76-10655b7382ba-istio-token\" (UniqueName: \"kubernetes.io/projected/428b29c0-c277-4e39-ab76-10655b7382ba-istio-token\") pod \"428b29c0-c277-4e39-ab76-10655b7382ba\" (UID: \"428b29c0-c277-4e39-ab76-10655b7382ba\") "
Oct 30 13:54:07.285700 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:07.285672    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/428b29c0-c277-4e39-ab76-10655b7382ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/428b29c0-c277-4e39-ab76-10655b7382ba-istio-podinfo\") pod \"428b29c0-c277-4e39-ab76-10655b7382ba\" (UID: \"428b29c0-c277-4e39-ab76-10655b7382ba\") "
Oct 30 13:54:07.285756 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:07.285700    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/428b29c0-c277-4e39-ab76-10655b7382ba-credential-socket" pod "428b29c0-c277-4e39-ab76-10655b7382ba" (UID: "428b29c0-c277-4e39-ab76-10655b7382ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:07.285756 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:07.285728    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/428b29c0-c277-4e39-ab76-10655b7382ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/428b29c0-c277-4e39-ab76-10655b7382ba-istiod-ca-cert\") pod \"428b29c0-c277-4e39-ab76-10655b7382ba\" (UID: \"428b29c0-c277-4e39-ab76-10655b7382ba\") "
Oct 30 13:54:07.285853 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:07.285826    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/428b29c0-c277-4e39-ab76-10655b7382ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/428b29c0-c277-4e39-ab76-10655b7382ba-workload-certs\") pod \"428b29c0-c277-4e39-ab76-10655b7382ba\" (UID: \"428b29c0-c277-4e39-ab76-10655b7382ba\") "
Oct 30 13:54:07.286210 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:07.286179    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/428b29c0-c277-4e39-ab76-10655b7382ba-istiod-ca-cert" pod "428b29c0-c277-4e39-ab76-10655b7382ba" (UID: "428b29c0-c277-4e39-ab76-10655b7382ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:54:07.286469 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:07.286443    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/428b29c0-c277-4e39-ab76-10655b7382ba-workload-certs" pod "428b29c0-c277-4e39-ab76-10655b7382ba" (UID: "428b29c0-c277-4e39-ab76-10655b7382ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:07.286652 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:07.286629    5093 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/428b29c0-c277-4e39-ab76-10655b7382ba-istiod-ca-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:54:07.286723 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:07.286658    5093 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/428b29c0-c277-4e39-ab76-10655b7382ba-workload-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:54:07.286723 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:07.286673    5093 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/428b29c0-c277-4e39-ab76-10655b7382ba-istio-data\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:54:07.286723 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:07.286686    5093 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/428b29c0-c277-4e39-ab76-10655b7382ba-workload-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:54:07.286723 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:07.286699    5093 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/428b29c0-c277-4e39-ab76-10655b7382ba-credential-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:54:07.287840 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:07.287810    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/428b29c0-c277-4e39-ab76-10655b7382ba-istio-token" pod "428b29c0-c277-4e39-ab76-10655b7382ba" (UID: "428b29c0-c277-4e39-ab76-10655b7382ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:54:07.287929 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:07.287810    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/428b29c0-c277-4e39-ab76-10655b7382ba-istio-envoy" pod "428b29c0-c277-4e39-ab76-10655b7382ba" (UID: "428b29c0-c277-4e39-ab76-10655b7382ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:07.287929 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:07.287868    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/428b29c0-c277-4e39-ab76-10655b7382ba-istio-podinfo" pod "428b29c0-c277-4e39-ab76-10655b7382ba" (UID: "428b29c0-c277-4e39-ab76-10655b7382ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:54:07.287929 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:07.287918    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/428b29c0-c277-4e39-ab76-10655b7382ba-kube-api-access-vmmrl" pod "428b29c0-c277-4e39-ab76-10655b7382ba" (UID: "428b29c0-c277-4e39-ab76-10655b7382ba"). InnerVolumeSpecName "kube-api-access-vmmrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:54:07.388621 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:07.388579    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/428b29c0-c277-4e39-ab76-10655b7382ba-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:54:07.388621 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:07.388613    5093 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/428b29c0-c277-4e39-ab76-10655b7382ba-istio-podinfo\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:54:07.388621 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:07.388624    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmmrl\" (UniqueName: \"kubernetes.io/projected/428b29c0-c277-4e39-ab76-10655b7382ba-kube-api-access-vmmrl\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:54:07.388869 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:07.388637    5093 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/428b29c0-c277-4e39-ab76-10655b7382ba-istio-envoy\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:54:07.633063 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:07.632953    5093 generic.go:356] "Generic (PLEG): container finished" podID="428b29c0-c277-4e39-ab76-10655b7382ba" containerID="105319a27172cc8133bb2aaee44b096dc497b2ea3db368c2373fb0939178a819" exitCode=0
Oct 30 13:54:07.638634 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:07.633222    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8"
Oct 30 13:54:07.638634 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:07.633244    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8" event={"ID":"428b29c0-c277-4e39-ab76-10655b7382ba","Type":"ContainerDied","Data":"105319a27172cc8133bb2aaee44b096dc497b2ea3db368c2373fb0939178a819"}
Oct 30 13:54:07.638634 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:07.635639    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8" event={"ID":"428b29c0-c277-4e39-ab76-10655b7382ba","Type":"ContainerDied","Data":"8307294254d0433a2e0be356147c0ba9f505943dda9e995dfe016364b27dd7d4"}
Oct 30 13:54:07.638634 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:07.635686    5093 scope.go:122] "RemoveContainer" containerID="105319a27172cc8133bb2aaee44b096dc497b2ea3db368c2373fb0939178a819"
Oct 30 13:54:07.671310 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:07.671286    5093 scope.go:122] "RemoveContainer" containerID="105319a27172cc8133bb2aaee44b096dc497b2ea3db368c2373fb0939178a819"
Oct 30 13:54:07.671696 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:54:07.671668    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"105319a27172cc8133bb2aaee44b096dc497b2ea3db368c2373fb0939178a819\": container with ID starting with 105319a27172cc8133bb2aaee44b096dc497b2ea3db368c2373fb0939178a819 not found: ID does not exist" containerID="105319a27172cc8133bb2aaee44b096dc497b2ea3db368c2373fb0939178a819"
Oct 30 13:54:07.671745 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:07.671709    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"105319a27172cc8133bb2aaee44b096dc497b2ea3db368c2373fb0939178a819"} err="failed to get container status \"105319a27172cc8133bb2aaee44b096dc497b2ea3db368c2373fb0939178a819\": rpc error: code = NotFound desc = could not find container \"105319a27172cc8133bb2aaee44b096dc497b2ea3db368c2373fb0939178a819\": container with ID starting with 105319a27172cc8133bb2aaee44b096dc497b2ea3db368c2373fb0939178a819 not found: ID does not exist"
Oct 30 13:54:07.705433 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:07.705399    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8"]
Oct 30 13:54:07.710407 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:07.710377    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-22af91-openshift-default-75dfd4c467-lnmr8"]
Oct 30 13:54:09.004476 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:09.004442    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="428b29c0-c277-4e39-ab76-10655b7382ba" path="/var/lib/kubelet/pods/428b29c0-c277-4e39-ab76-10655b7382ba/volumes"
Oct 30 13:54:16.395203 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:16.395171    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-08c365/test-model-08c365-kserve-576d588df-fsfnh"]
Oct 30 13:54:16.395993 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:16.395979    5093 cpu_manager.go:414] "containerMap: removing container" podUID="73976ccb-1e12-40ef-bf4b-6c5deeda3b88" containerName="sim"
Oct 30 13:54:16.396042 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:16.395995    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73976ccb-1e12-40ef-bf4b-6c5deeda3b88" containerName="sim"
Oct 30 13:54:16.396042 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:16.396027    5093 cpu_manager.go:414] "containerMap: removing container" podUID="428b29c0-c277-4e39-ab76-10655b7382ba" containerName="istio-proxy"
Oct 30 13:54:16.396042 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:16.396033    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="428b29c0-c277-4e39-ab76-10655b7382ba" containerName="istio-proxy"
Oct 30 13:54:16.396203 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:16.396190    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="73976ccb-1e12-40ef-bf4b-6c5deeda3b88" containerName="sim"
Oct 30 13:54:16.396246 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:16.396208    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="428b29c0-c277-4e39-ab76-10655b7382ba" containerName="istio-proxy"
Oct 30 13:54:16.401688 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:16.401661    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-08c365/test-model-08c365-kserve-576d588df-fsfnh"
Oct 30 13:54:16.424157 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:16.424105    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-08c365/test-model-08c365-kserve-576d588df-fsfnh"]
Oct 30 13:54:16.429999 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:16.429963    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-08c365\"/\"openshift-service-ca.crt\""
Oct 30 13:54:16.430273 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:16.430247    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-08c365\"/\"kube-root-ca.crt\""
Oct 30 13:54:16.430500 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:16.430481    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-08c365\"/\"test-model-08c365-kserve-self-signed-certs\""
Oct 30 13:54:16.430784 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:16.430765    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-08c365\"/\"default-dockercfg-lf2rm\""
Oct 30 13:54:16.545907 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:16.545868    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f73d682-50c2-46d3-a907-dc4c6fb089af-dshm\") pod \"test-model-08c365-kserve-576d588df-fsfnh\" (UID: \"2f73d682-50c2-46d3-a907-dc4c6fb089af\") " pod="ai-tenant-e2e-inf-a-08c365/test-model-08c365-kserve-576d588df-fsfnh"
Oct 30 13:54:16.546097 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:16.546037    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f73d682-50c2-46d3-a907-dc4c6fb089af-tls-certs\") pod \"test-model-08c365-kserve-576d588df-fsfnh\" (UID: \"2f73d682-50c2-46d3-a907-dc4c6fb089af\") " pod="ai-tenant-e2e-inf-a-08c365/test-model-08c365-kserve-576d588df-fsfnh"
Oct 30 13:54:16.546250 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:16.546227    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45nn9\" (UniqueName: \"kubernetes.io/projected/2f73d682-50c2-46d3-a907-dc4c6fb089af-kube-api-access-45nn9\") pod \"test-model-08c365-kserve-576d588df-fsfnh\" (UID: \"2f73d682-50c2-46d3-a907-dc4c6fb089af\") " pod="ai-tenant-e2e-inf-a-08c365/test-model-08c365-kserve-576d588df-fsfnh"
Oct 30 13:54:16.546327 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:16.546267    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f73d682-50c2-46d3-a907-dc4c6fb089af-model-cache\") pod \"test-model-08c365-kserve-576d588df-fsfnh\" (UID: \"2f73d682-50c2-46d3-a907-dc4c6fb089af\") " pod="ai-tenant-e2e-inf-a-08c365/test-model-08c365-kserve-576d588df-fsfnh"
Oct 30 13:54:16.546388 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:16.546337    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f73d682-50c2-46d3-a907-dc4c6fb089af-tmp-dir\") pod \"test-model-08c365-kserve-576d588df-fsfnh\" (UID: \"2f73d682-50c2-46d3-a907-dc4c6fb089af\") " pod="ai-tenant-e2e-inf-a-08c365/test-model-08c365-kserve-576d588df-fsfnh"
Oct 30 13:54:16.546518 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:16.546500    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f73d682-50c2-46d3-a907-dc4c6fb089af-home\") pod \"test-model-08c365-kserve-576d588df-fsfnh\" (UID: \"2f73d682-50c2-46d3-a907-dc4c6fb089af\") " pod="ai-tenant-e2e-inf-a-08c365/test-model-08c365-kserve-576d588df-fsfnh"
Oct 30 13:54:16.661207 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:16.652194    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f73d682-50c2-46d3-a907-dc4c6fb089af-dshm\") pod \"test-model-08c365-kserve-576d588df-fsfnh\" (UID: \"2f73d682-50c2-46d3-a907-dc4c6fb089af\") " pod="ai-tenant-e2e-inf-a-08c365/test-model-08c365-kserve-576d588df-fsfnh"
Oct 30 13:54:16.661207 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:16.652385    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f73d682-50c2-46d3-a907-dc4c6fb089af-tls-certs\") pod \"test-model-08c365-kserve-576d588df-fsfnh\" (UID: \"2f73d682-50c2-46d3-a907-dc4c6fb089af\") " pod="ai-tenant-e2e-inf-a-08c365/test-model-08c365-kserve-576d588df-fsfnh"
Oct 30 13:54:16.661207 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:16.652655    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45nn9\" (UniqueName: \"kubernetes.io/projected/2f73d682-50c2-46d3-a907-dc4c6fb089af-kube-api-access-45nn9\") pod \"test-model-08c365-kserve-576d588df-fsfnh\" (UID: \"2f73d682-50c2-46d3-a907-dc4c6fb089af\") " pod="ai-tenant-e2e-inf-a-08c365/test-model-08c365-kserve-576d588df-fsfnh"
Oct 30 13:54:16.661207 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:16.652687    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f73d682-50c2-46d3-a907-dc4c6fb089af-model-cache\") pod \"test-model-08c365-kserve-576d588df-fsfnh\" (UID: \"2f73d682-50c2-46d3-a907-dc4c6fb089af\") " pod="ai-tenant-e2e-inf-a-08c365/test-model-08c365-kserve-576d588df-fsfnh"
Oct 30 13:54:16.661207 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:16.652716    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f73d682-50c2-46d3-a907-dc4c6fb089af-tmp-dir\") pod \"test-model-08c365-kserve-576d588df-fsfnh\" (UID: \"2f73d682-50c2-46d3-a907-dc4c6fb089af\") " pod="ai-tenant-e2e-inf-a-08c365/test-model-08c365-kserve-576d588df-fsfnh"
Oct 30 13:54:16.661207 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:16.652857    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f73d682-50c2-46d3-a907-dc4c6fb089af-home\") pod \"test-model-08c365-kserve-576d588df-fsfnh\" (UID: \"2f73d682-50c2-46d3-a907-dc4c6fb089af\") " pod="ai-tenant-e2e-inf-a-08c365/test-model-08c365-kserve-576d588df-fsfnh"
Oct 30 13:54:16.661207 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:16.653397    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f73d682-50c2-46d3-a907-dc4c6fb089af-home\") pod \"test-model-08c365-kserve-576d588df-fsfnh\" (UID: \"2f73d682-50c2-46d3-a907-dc4c6fb089af\") " pod="ai-tenant-e2e-inf-a-08c365/test-model-08c365-kserve-576d588df-fsfnh"
Oct 30 13:54:16.663479 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:16.663449    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f73d682-50c2-46d3-a907-dc4c6fb089af-model-cache\") pod \"test-model-08c365-kserve-576d588df-fsfnh\" (UID: \"2f73d682-50c2-46d3-a907-dc4c6fb089af\") " pod="ai-tenant-e2e-inf-a-08c365/test-model-08c365-kserve-576d588df-fsfnh"
Oct 30 13:54:16.685820 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:16.663906    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f73d682-50c2-46d3-a907-dc4c6fb089af-tmp-dir\") pod \"test-model-08c365-kserve-576d588df-fsfnh\" (UID: \"2f73d682-50c2-46d3-a907-dc4c6fb089af\") " pod="ai-tenant-e2e-inf-a-08c365/test-model-08c365-kserve-576d588df-fsfnh"
Oct 30 13:54:16.706152 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:16.703761    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f73d682-50c2-46d3-a907-dc4c6fb089af-tls-certs\") pod \"test-model-08c365-kserve-576d588df-fsfnh\" (UID: \"2f73d682-50c2-46d3-a907-dc4c6fb089af\") " pod="ai-tenant-e2e-inf-a-08c365/test-model-08c365-kserve-576d588df-fsfnh"
Oct 30 13:54:16.740563 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:16.735554    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45nn9\" (UniqueName: \"kubernetes.io/projected/2f73d682-50c2-46d3-a907-dc4c6fb089af-kube-api-access-45nn9\") pod \"test-model-08c365-kserve-576d588df-fsfnh\" (UID: \"2f73d682-50c2-46d3-a907-dc4c6fb089af\") " pod="ai-tenant-e2e-inf-a-08c365/test-model-08c365-kserve-576d588df-fsfnh"
Oct 30 13:54:16.763776 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:16.763740    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f73d682-50c2-46d3-a907-dc4c6fb089af-dshm\") pod \"test-model-08c365-kserve-576d588df-fsfnh\" (UID: \"2f73d682-50c2-46d3-a907-dc4c6fb089af\") " pod="ai-tenant-e2e-inf-a-08c365/test-model-08c365-kserve-576d588df-fsfnh"
Oct 30 13:54:17.025099 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:17.025062    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-08c365/test-model-08c365-kserve-576d588df-fsfnh"
Oct 30 13:54:17.348893 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:17.348859    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-08c365/test-model-08c365-kserve-576d588df-fsfnh"]
Oct 30 13:54:17.349428 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:54:17.349401    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f73d682_50c2_46d3_a907_dc4c6fb089af.slice/crio-c212b7e9fc5ac5fe787ed3c7318fae83dc1740f994c7024db80959247b547f17 WatchSource:0}: Error finding container c212b7e9fc5ac5fe787ed3c7318fae83dc1740f994c7024db80959247b547f17: Status 404 returned error can't find the container with id c212b7e9fc5ac5fe787ed3c7318fae83dc1740f994c7024db80959247b547f17
Oct 30 13:54:17.773249 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:17.773213    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-08c365/test-model-08c365-kserve-576d588df-fsfnh" event={"ID":"2f73d682-50c2-46d3-a907-dc4c6fb089af","Type":"ContainerStarted","Data":"f73f4c35aacad6329ca64a4a2f639d95cb54b06b6b9848d8ae2e55d0fca3060c"}
Oct 30 13:54:17.774058 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:17.773263    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-08c365/test-model-08c365-kserve-576d588df-fsfnh" event={"ID":"2f73d682-50c2-46d3-a907-dc4c6fb089af","Type":"ContainerStarted","Data":"c212b7e9fc5ac5fe787ed3c7318fae83dc1740f994c7024db80959247b547f17"}
Oct 30 13:54:17.817442 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:17.817390    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-08c365/test-model-08c365-kserve-576d588df-fsfnh" podStartSLOduration=1.817376097 podStartE2EDuration="1.817376097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:54:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:54:17.813992553 +0000 UTC m=+3076.961259794" watchObservedRunningTime="2026-10-30 13:54:17.817376097 +0000 UTC m=+3076.964643338"
Oct 30 13:54:18.025991 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:18.025897    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-08c365/test-model-08c365-kserve-576d588df-fsfnh"
Oct 30 13:54:21.176824 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:54:21.176774    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:54:27.026046 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:27.026003    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-08c365/test-model-08c365-kserve-576d588df-fsfnh"
Oct 30 13:54:27.040300 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:27.040261    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-08c365/test-model-08c365-kserve-576d588df-fsfnh"
Oct 30 13:54:27.917934 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:27.917897    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-08c365/test-model-08c365-kserve-576d588df-fsfnh"
Oct 30 13:54:48.868436 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:48.868340    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d724c8-54d88bfbbb-t6j9r"]
Oct 30 13:54:48.873722 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:48.873694    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d724c8-54d88bfbbb-t6j9r"
Oct 30 13:54:48.893346 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:48.893313    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d724c8-54d88bfbbb-t6j9r"]
Oct 30 13:54:48.896449 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:48.896418    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-d724c8-serving-cert\""
Oct 30 13:54:48.952080 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:48.952041    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b676aeef-6a09-4e04-929d-6388b8034eac-maas-api-tls\") pod \"maas-api-e2e-inf-a-d724c8-54d88bfbbb-t6j9r\" (UID: \"b676aeef-6a09-4e04-929d-6388b8034eac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d724c8-54d88bfbbb-t6j9r"
Oct 30 13:54:48.952347 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:48.952201    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b676aeef-6a09-4e04-929d-6388b8034eac-metrics-certs\") pod \"maas-api-e2e-inf-a-d724c8-54d88bfbbb-t6j9r\" (UID: \"b676aeef-6a09-4e04-929d-6388b8034eac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d724c8-54d88bfbbb-t6j9r"
Oct 30 13:54:48.952347 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:48.952239    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdvnj\" (UniqueName: \"kubernetes.io/projected/b676aeef-6a09-4e04-929d-6388b8034eac-kube-api-access-qdvnj\") pod \"maas-api-e2e-inf-a-d724c8-54d88bfbbb-t6j9r\" (UID: \"b676aeef-6a09-4e04-929d-6388b8034eac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d724c8-54d88bfbbb-t6j9r"
Oct 30 13:54:48.952347 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:48.952311    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b676aeef-6a09-4e04-929d-6388b8034eac-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d724c8-54d88bfbbb-t6j9r\" (UID: \"b676aeef-6a09-4e04-929d-6388b8034eac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d724c8-54d88bfbbb-t6j9r"
Oct 30 13:54:49.053791 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:49.053749    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b676aeef-6a09-4e04-929d-6388b8034eac-maas-api-tls\") pod \"maas-api-e2e-inf-a-d724c8-54d88bfbbb-t6j9r\" (UID: \"b676aeef-6a09-4e04-929d-6388b8034eac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d724c8-54d88bfbbb-t6j9r"
Oct 30 13:54:49.054011 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:49.053814    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b676aeef-6a09-4e04-929d-6388b8034eac-metrics-certs\") pod \"maas-api-e2e-inf-a-d724c8-54d88bfbbb-t6j9r\" (UID: \"b676aeef-6a09-4e04-929d-6388b8034eac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d724c8-54d88bfbbb-t6j9r"
Oct 30 13:54:49.054011 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:49.053848    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdvnj\" (UniqueName: \"kubernetes.io/projected/b676aeef-6a09-4e04-929d-6388b8034eac-kube-api-access-qdvnj\") pod \"maas-api-e2e-inf-a-d724c8-54d88bfbbb-t6j9r\" (UID: \"b676aeef-6a09-4e04-929d-6388b8034eac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d724c8-54d88bfbbb-t6j9r"
Oct 30 13:54:49.054011 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:49.053985    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b676aeef-6a09-4e04-929d-6388b8034eac-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d724c8-54d88bfbbb-t6j9r\" (UID: \"b676aeef-6a09-4e04-929d-6388b8034eac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d724c8-54d88bfbbb-t6j9r"
Oct 30 13:54:49.055024 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:49.054986    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b676aeef-6a09-4e04-929d-6388b8034eac-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d724c8-54d88bfbbb-t6j9r\" (UID: \"b676aeef-6a09-4e04-929d-6388b8034eac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d724c8-54d88bfbbb-t6j9r"
Oct 30 13:54:49.056685 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:49.056653    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b676aeef-6a09-4e04-929d-6388b8034eac-metrics-certs\") pod \"maas-api-e2e-inf-a-d724c8-54d88bfbbb-t6j9r\" (UID: \"b676aeef-6a09-4e04-929d-6388b8034eac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d724c8-54d88bfbbb-t6j9r"
Oct 30 13:54:49.056841 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:49.056793    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b676aeef-6a09-4e04-929d-6388b8034eac-maas-api-tls\") pod \"maas-api-e2e-inf-a-d724c8-54d88bfbbb-t6j9r\" (UID: \"b676aeef-6a09-4e04-929d-6388b8034eac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d724c8-54d88bfbbb-t6j9r"
Oct 30 13:54:49.077118 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:49.077081    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdvnj\" (UniqueName: \"kubernetes.io/projected/b676aeef-6a09-4e04-929d-6388b8034eac-kube-api-access-qdvnj\") pod \"maas-api-e2e-inf-a-d724c8-54d88bfbbb-t6j9r\" (UID: \"b676aeef-6a09-4e04-929d-6388b8034eac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d724c8-54d88bfbbb-t6j9r"
Oct 30 13:54:49.196089 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:49.196052    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d724c8-54d88bfbbb-t6j9r"
Oct 30 13:54:49.516292 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:49.516237    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d724c8-54d88bfbbb-t6j9r"]
Oct 30 13:54:49.516633 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:54:49.516602    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb676aeef_6a09_4e04_929d_6388b8034eac.slice/crio-f1f1c6fbf2018c0008b83e97d3bfa577cd5bf3c1a62cc99bbe6588ce149566e9 WatchSource:0}: Error finding container f1f1c6fbf2018c0008b83e97d3bfa577cd5bf3c1a62cc99bbe6588ce149566e9: Status 404 returned error can't find the container with id f1f1c6fbf2018c0008b83e97d3bfa577cd5bf3c1a62cc99bbe6588ce149566e9
Oct 30 13:54:50.127655 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:50.126701    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d724c8-54d88bfbbb-t6j9r" event={"ID":"b676aeef-6a09-4e04-929d-6388b8034eac","Type":"ContainerStarted","Data":"7f3264e45a87ad8670a358ea9cd775157ff1305b2bd76c0def97bc5fdb44905a"}
Oct 30 13:54:50.127655 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:50.126759    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d724c8-54d88bfbbb-t6j9r" event={"ID":"b676aeef-6a09-4e04-929d-6388b8034eac","Type":"ContainerStarted","Data":"f1f1c6fbf2018c0008b83e97d3bfa577cd5bf3c1a62cc99bbe6588ce149566e9"}
Oct 30 13:54:50.127655 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:50.127327    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d724c8-54d88bfbbb-t6j9r"
Oct 30 13:54:50.171713 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:50.171663    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d724c8-54d88bfbbb-t6j9r" podStartSLOduration=1.7693405979999999 podStartE2EDuration="2.171644074s" totalImagesPullingTime="402.303476ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:54:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:54:50.169449225 +0000 UTC m=+3109.316716468" watchObservedRunningTime="2026-10-30 13:54:50.171644074 +0000 UTC m=+3109.318911314"
Oct 30 13:54:51.174945 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:54:51.174894    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:54:56.144643 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:54:56.144603    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d724c8-54d88bfbbb-t6j9r"
Oct 30 13:55:16.490737 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:16.490695    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-926ff0/test-model-926ff0-kserve-84bbd495b5-x2fw2"]
Oct 30 13:55:16.497233 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:16.497207    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-926ff0/test-model-926ff0-kserve-84bbd495b5-x2fw2"
Oct 30 13:55:16.527309 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:16.527274    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-926ff0\"/\"openshift-service-ca.crt\""
Oct 30 13:55:16.527763 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:16.527632    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-926ff0\"/\"kube-root-ca.crt\""
Oct 30 13:55:16.527763 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:16.527729    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-926ff0\"/\"test-model-926ff0-kserve-self-signed-certs\""
Oct 30 13:55:16.529880 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:16.529855    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-926ff0\"/\"default-dockercfg-klrcv\""
Oct 30 13:55:16.535699 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:16.535666    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-926ff0/test-model-926ff0-kserve-84bbd495b5-x2fw2"]
Oct 30 13:55:16.671144 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:16.671084    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37c1896c-d703-40c3-8be8-94ced507e3e0-dshm\") pod \"test-model-926ff0-kserve-84bbd495b5-x2fw2\" (UID: \"37c1896c-d703-40c3-8be8-94ced507e3e0\") " pod="ai-tenant-e2e-inf-a-926ff0/test-model-926ff0-kserve-84bbd495b5-x2fw2"
Oct 30 13:55:16.671362 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:16.671203    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37c1896c-d703-40c3-8be8-94ced507e3e0-tls-certs\") pod \"test-model-926ff0-kserve-84bbd495b5-x2fw2\" (UID: \"37c1896c-d703-40c3-8be8-94ced507e3e0\") " pod="ai-tenant-e2e-inf-a-926ff0/test-model-926ff0-kserve-84bbd495b5-x2fw2"
Oct 30 13:55:16.671362 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:16.671253    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37c1896c-d703-40c3-8be8-94ced507e3e0-home\") pod \"test-model-926ff0-kserve-84bbd495b5-x2fw2\" (UID: \"37c1896c-d703-40c3-8be8-94ced507e3e0\") " pod="ai-tenant-e2e-inf-a-926ff0/test-model-926ff0-kserve-84bbd495b5-x2fw2"
Oct 30 13:55:16.671490 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:16.671450    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37c1896c-d703-40c3-8be8-94ced507e3e0-model-cache\") pod \"test-model-926ff0-kserve-84bbd495b5-x2fw2\" (UID: \"37c1896c-d703-40c3-8be8-94ced507e3e0\") " pod="ai-tenant-e2e-inf-a-926ff0/test-model-926ff0-kserve-84bbd495b5-x2fw2"
Oct 30 13:55:16.671552 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:16.671509    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgk9q\" (UniqueName: \"kubernetes.io/projected/37c1896c-d703-40c3-8be8-94ced507e3e0-kube-api-access-wgk9q\") pod \"test-model-926ff0-kserve-84bbd495b5-x2fw2\" (UID: \"37c1896c-d703-40c3-8be8-94ced507e3e0\") " pod="ai-tenant-e2e-inf-a-926ff0/test-model-926ff0-kserve-84bbd495b5-x2fw2"
Oct 30 13:55:16.671621 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:16.671573    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37c1896c-d703-40c3-8be8-94ced507e3e0-tmp-dir\") pod \"test-model-926ff0-kserve-84bbd495b5-x2fw2\" (UID: \"37c1896c-d703-40c3-8be8-94ced507e3e0\") " pod="ai-tenant-e2e-inf-a-926ff0/test-model-926ff0-kserve-84bbd495b5-x2fw2"
Oct 30 13:55:16.779202 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:16.773184    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37c1896c-d703-40c3-8be8-94ced507e3e0-home\") pod \"test-model-926ff0-kserve-84bbd495b5-x2fw2\" (UID: \"37c1896c-d703-40c3-8be8-94ced507e3e0\") " pod="ai-tenant-e2e-inf-a-926ff0/test-model-926ff0-kserve-84bbd495b5-x2fw2"
Oct 30 13:55:16.779202 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:16.773371    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37c1896c-d703-40c3-8be8-94ced507e3e0-model-cache\") pod \"test-model-926ff0-kserve-84bbd495b5-x2fw2\" (UID: \"37c1896c-d703-40c3-8be8-94ced507e3e0\") " pod="ai-tenant-e2e-inf-a-926ff0/test-model-926ff0-kserve-84bbd495b5-x2fw2"
Oct 30 13:55:16.779202 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:16.773453    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgk9q\" (UniqueName: \"kubernetes.io/projected/37c1896c-d703-40c3-8be8-94ced507e3e0-kube-api-access-wgk9q\") pod \"test-model-926ff0-kserve-84bbd495b5-x2fw2\" (UID: \"37c1896c-d703-40c3-8be8-94ced507e3e0\") " pod="ai-tenant-e2e-inf-a-926ff0/test-model-926ff0-kserve-84bbd495b5-x2fw2"
Oct 30 13:55:16.779202 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:16.773530    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37c1896c-d703-40c3-8be8-94ced507e3e0-tmp-dir\") pod \"test-model-926ff0-kserve-84bbd495b5-x2fw2\" (UID: \"37c1896c-d703-40c3-8be8-94ced507e3e0\") " pod="ai-tenant-e2e-inf-a-926ff0/test-model-926ff0-kserve-84bbd495b5-x2fw2"
Oct 30 13:55:16.779202 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:16.773775    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37c1896c-d703-40c3-8be8-94ced507e3e0-dshm\") pod \"test-model-926ff0-kserve-84bbd495b5-x2fw2\" (UID: \"37c1896c-d703-40c3-8be8-94ced507e3e0\") " pod="ai-tenant-e2e-inf-a-926ff0/test-model-926ff0-kserve-84bbd495b5-x2fw2"
Oct 30 13:55:16.779202 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:16.773874    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37c1896c-d703-40c3-8be8-94ced507e3e0-tls-certs\") pod \"test-model-926ff0-kserve-84bbd495b5-x2fw2\" (UID: \"37c1896c-d703-40c3-8be8-94ced507e3e0\") " pod="ai-tenant-e2e-inf-a-926ff0/test-model-926ff0-kserve-84bbd495b5-x2fw2"
Oct 30 13:55:16.780084 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:16.780059    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37c1896c-d703-40c3-8be8-94ced507e3e0-model-cache\") pod \"test-model-926ff0-kserve-84bbd495b5-x2fw2\" (UID: \"37c1896c-d703-40c3-8be8-94ced507e3e0\") " pod="ai-tenant-e2e-inf-a-926ff0/test-model-926ff0-kserve-84bbd495b5-x2fw2"
Oct 30 13:55:16.780639 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:16.780616    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37c1896c-d703-40c3-8be8-94ced507e3e0-home\") pod \"test-model-926ff0-kserve-84bbd495b5-x2fw2\" (UID: \"37c1896c-d703-40c3-8be8-94ced507e3e0\") " pod="ai-tenant-e2e-inf-a-926ff0/test-model-926ff0-kserve-84bbd495b5-x2fw2"
Oct 30 13:55:16.781916 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:16.781608    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37c1896c-d703-40c3-8be8-94ced507e3e0-tmp-dir\") pod \"test-model-926ff0-kserve-84bbd495b5-x2fw2\" (UID: \"37c1896c-d703-40c3-8be8-94ced507e3e0\") " pod="ai-tenant-e2e-inf-a-926ff0/test-model-926ff0-kserve-84bbd495b5-x2fw2"
Oct 30 13:55:16.812345 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:16.812314    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37c1896c-d703-40c3-8be8-94ced507e3e0-tls-certs\") pod \"test-model-926ff0-kserve-84bbd495b5-x2fw2\" (UID: \"37c1896c-d703-40c3-8be8-94ced507e3e0\") " pod="ai-tenant-e2e-inf-a-926ff0/test-model-926ff0-kserve-84bbd495b5-x2fw2"
Oct 30 13:55:16.829162 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:16.816481    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37c1896c-d703-40c3-8be8-94ced507e3e0-dshm\") pod \"test-model-926ff0-kserve-84bbd495b5-x2fw2\" (UID: \"37c1896c-d703-40c3-8be8-94ced507e3e0\") " pod="ai-tenant-e2e-inf-a-926ff0/test-model-926ff0-kserve-84bbd495b5-x2fw2"
Oct 30 13:55:16.829162 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:16.817380    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgk9q\" (UniqueName: \"kubernetes.io/projected/37c1896c-d703-40c3-8be8-94ced507e3e0-kube-api-access-wgk9q\") pod \"test-model-926ff0-kserve-84bbd495b5-x2fw2\" (UID: \"37c1896c-d703-40c3-8be8-94ced507e3e0\") " pod="ai-tenant-e2e-inf-a-926ff0/test-model-926ff0-kserve-84bbd495b5-x2fw2"
Oct 30 13:55:17.111927 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:17.111829    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-926ff0/test-model-926ff0-kserve-84bbd495b5-x2fw2"
Oct 30 13:55:17.439491 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:17.438859    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-926ff0/test-model-926ff0-kserve-84bbd495b5-x2fw2"]
Oct 30 13:55:17.442952 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:55:17.442900    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37c1896c_d703_40c3_8be8_94ced507e3e0.slice/crio-5dd093c0b98e71cfb4d4f093c1e35079cf13867b2178dc2293e91281ec59675f WatchSource:0}: Error finding container 5dd093c0b98e71cfb4d4f093c1e35079cf13867b2178dc2293e91281ec59675f: Status 404 returned error can't find the container with id 5dd093c0b98e71cfb4d4f093c1e35079cf13867b2178dc2293e91281ec59675f
Oct 30 13:55:17.463181 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:17.462713    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-926ff0/test-model-926ff0-kserve-84bbd495b5-x2fw2" event={"ID":"37c1896c-d703-40c3-8be8-94ced507e3e0","Type":"ContainerStarted","Data":"5dd093c0b98e71cfb4d4f093c1e35079cf13867b2178dc2293e91281ec59675f"}
Oct 30 13:55:18.477419 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:18.477380    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-926ff0/test-model-926ff0-kserve-84bbd495b5-x2fw2" event={"ID":"37c1896c-d703-40c3-8be8-94ced507e3e0","Type":"ContainerStarted","Data":"efb3c819d4c8b2eb3d758060d673d2f1831f9e69377f68e0ac0d6d681e1b6889"}
Oct 30 13:55:18.518229 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:18.518160    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-926ff0/test-model-926ff0-kserve-84bbd495b5-x2fw2" podStartSLOduration=2.518142306 podStartE2EDuration="2.518142306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:55:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:55:18.516925882 +0000 UTC m=+3137.664193136" watchObservedRunningTime="2026-10-30 13:55:18.518142306 +0000 UTC m=+3137.665409540"
Oct 30 13:55:19.111968 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:19.111933    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-926ff0/test-model-926ff0-kserve-84bbd495b5-x2fw2"
Oct 30 13:55:21.169371 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:55:21.169328    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:55:27.112224 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:27.112168    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-926ff0/test-model-926ff0-kserve-84bbd495b5-x2fw2"
Oct 30 13:55:27.127613 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:27.127574    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-926ff0/test-model-926ff0-kserve-84bbd495b5-x2fw2"
Oct 30 13:55:27.593025 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:27.592993    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-926ff0/test-model-926ff0-kserve-84bbd495b5-x2fw2"
Oct 30 13:55:34.766696 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:34.766654    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft"]
Oct 30 13:55:34.767212 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:34.766865    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft" podUID="aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8" containerName="istio-proxy" containerID="cri-o://78d7dc53d6bd6c760b98468350d1fa315f64499ada0c47bd00ab8a24011f31c8" gracePeriod=30
Oct 30 13:55:39.445307 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:39.445265    5093 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:55:39.445838 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:39.445348    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft" podUID="aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:55:40.072044 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:40.072016    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft"
Oct 30 13:55:40.181390 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:40.181355    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-kube-api-access-pw6bs\" (UniqueName: \"kubernetes.io/projected/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-kube-api-access-pw6bs\") pod \"aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8\" (UID: \"aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8\") "
Oct 30 13:55:40.181606 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:40.181446    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-workload-socket\") pod \"aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8\" (UID: \"aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8\") "
Oct 30 13:55:40.181606 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:40.181551    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-istiod-ca-cert\") pod \"aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8\" (UID: \"aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8\") "
Oct 30 13:55:40.181738 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:40.181660    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-istio-envoy\") pod \"aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8\" (UID: \"aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8\") "
Oct 30 13:55:40.181798 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:40.181747    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-workload-certs\") pod \"aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8\" (UID: \"aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8\") "
Oct 30 13:55:40.181798 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:40.181770    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-workload-socket" pod "aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8" (UID: "aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:40.181929 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:40.181909    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-istio-data\") pod \"aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8\" (UID: \"aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8\") "
Oct 30 13:55:40.182161 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:40.182083    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-workload-certs" pod "aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8" (UID: "aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:40.182161 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:40.182097    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-istio-token\" (UniqueName: \"kubernetes.io/projected/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-istio-token\") pod \"aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8\" (UID: \"aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8\") "
Oct 30 13:55:40.182161 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:40.182151    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-istio-podinfo\") pod \"aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8\" (UID: \"aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8\") "
Oct 30 13:55:40.182790 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:40.182184    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-credential-socket\") pod \"aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8\" (UID: \"aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8\") "
Oct 30 13:55:40.182790 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:40.182335    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-istio-data" pod "aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8" (UID: "aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:40.182790 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:40.182634    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-credential-socket" pod "aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8" (UID: "aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:40.182973 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:40.182826    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-istiod-ca-cert" pod "aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8" (UID: "aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:55:40.183067 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:40.183041    5093 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-istio-data\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:55:40.183149 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:40.183073    5093 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-credential-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:55:40.183149 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:40.183088    5093 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-workload-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:55:40.183149 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:40.183105    5093 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-istiod-ca-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:55:40.183149 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:40.183118    5093 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-workload-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:55:40.184595 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:40.184562    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-istio-envoy" pod "aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8" (UID: "aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:40.185522 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:40.185490    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-istio-podinfo" pod "aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8" (UID: "aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:55:40.185667 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:40.185639    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-istio-token" pod "aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8" (UID: "aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:55:40.185719 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:40.185697    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-kube-api-access-pw6bs" pod "aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8" (UID: "aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8"). InnerVolumeSpecName "kube-api-access-pw6bs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:55:40.284941 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:40.284848    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:55:40.284941 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:40.284884    5093 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-istio-podinfo\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:55:40.284941 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:40.284893    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw6bs\" (UniqueName: \"kubernetes.io/projected/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-kube-api-access-pw6bs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:55:40.284941 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:40.284903    5093 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8-istio-envoy\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:55:40.714980 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:40.714941    5093 generic.go:356] "Generic (PLEG): container finished" podID="aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8" containerID="78d7dc53d6bd6c760b98468350d1fa315f64499ada0c47bd00ab8a24011f31c8" exitCode=0
Oct 30 13:55:40.715527 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:40.714986    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft" event={"ID":"aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8","Type":"ContainerDied","Data":"78d7dc53d6bd6c760b98468350d1fa315f64499ada0c47bd00ab8a24011f31c8"}
Oct 30 13:55:40.715527 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:40.715023    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft"
Oct 30 13:55:40.715527 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:40.715063    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft" event={"ID":"aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8","Type":"ContainerDied","Data":"4637e237d0f10102570ac15499752e872b1d628112582d8cedb157cb2e58b572"}
Oct 30 13:55:40.715527 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:40.715100    5093 scope.go:122] "RemoveContainer" containerID="78d7dc53d6bd6c760b98468350d1fa315f64499ada0c47bd00ab8a24011f31c8"
Oct 30 13:55:40.730166 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:40.730139    5093 scope.go:122] "RemoveContainer" containerID="78d7dc53d6bd6c760b98468350d1fa315f64499ada0c47bd00ab8a24011f31c8"
Oct 30 13:55:40.730558 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:55:40.730533    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78d7dc53d6bd6c760b98468350d1fa315f64499ada0c47bd00ab8a24011f31c8\": container with ID starting with 78d7dc53d6bd6c760b98468350d1fa315f64499ada0c47bd00ab8a24011f31c8 not found: ID does not exist" containerID="78d7dc53d6bd6c760b98468350d1fa315f64499ada0c47bd00ab8a24011f31c8"
Oct 30 13:55:40.730650 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:40.730568    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78d7dc53d6bd6c760b98468350d1fa315f64499ada0c47bd00ab8a24011f31c8"} err="failed to get container status \"78d7dc53d6bd6c760b98468350d1fa315f64499ada0c47bd00ab8a24011f31c8\": rpc error: code = NotFound desc = could not find container \"78d7dc53d6bd6c760b98468350d1fa315f64499ada0c47bd00ab8a24011f31c8\": container with ID starting with 78d7dc53d6bd6c760b98468350d1fa315f64499ada0c47bd00ab8a24011f31c8 not found: ID does not exist"
Oct 30 13:55:40.786514 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:40.786476    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft"]
Oct 30 13:55:40.799259 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:40.799218    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-bdkft"]
Oct 30 13:55:41.003511 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:41.003431    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8" path="/var/lib/kubelet/pods/aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8/volumes"
Oct 30 13:55:47.259396 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:47.259354    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c"]
Oct 30 13:55:47.259966 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:47.259683    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c" podUID="d6d8c234-2d8c-488b-9ea9-3969e9c430bf" containerName="istio-proxy" containerID="cri-o://87530e3ee9e3634a95376323c464547ab7e51a78d46c02c287a4c2af699a3d13" gracePeriod=30
Oct 30 13:55:51.171100 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:55:51.171058    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:55:52.690890 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:52.690858    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c"
Oct 30 13:55:52.728984 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:52.728943    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-istio-data\") pod \"d6d8c234-2d8c-488b-9ea9-3969e9c430bf\" (UID: \"d6d8c234-2d8c-488b-9ea9-3969e9c430bf\") "
Oct 30 13:55:52.729213 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:52.729060    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-kube-api-access-bsr6r\" (UniqueName: \"kubernetes.io/projected/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-kube-api-access-bsr6r\") pod \"d6d8c234-2d8c-488b-9ea9-3969e9c430bf\" (UID: \"d6d8c234-2d8c-488b-9ea9-3969e9c430bf\") "
Oct 30 13:55:52.729213 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:52.729118    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-istio-podinfo\") pod \"d6d8c234-2d8c-488b-9ea9-3969e9c430bf\" (UID: \"d6d8c234-2d8c-488b-9ea9-3969e9c430bf\") "
Oct 30 13:55:52.729327 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:52.729235    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-workload-socket\") pod \"d6d8c234-2d8c-488b-9ea9-3969e9c430bf\" (UID: \"d6d8c234-2d8c-488b-9ea9-3969e9c430bf\") "
Oct 30 13:55:52.729327 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:52.729263    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-istio-token\" (UniqueName: \"kubernetes.io/projected/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-istio-token\") pod \"d6d8c234-2d8c-488b-9ea9-3969e9c430bf\" (UID: \"d6d8c234-2d8c-488b-9ea9-3969e9c430bf\") "
Oct 30 13:55:52.729420 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:52.729336    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-istio-envoy\") pod \"d6d8c234-2d8c-488b-9ea9-3969e9c430bf\" (UID: \"d6d8c234-2d8c-488b-9ea9-3969e9c430bf\") "
Oct 30 13:55:52.729420 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:52.729380    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-credential-socket\") pod \"d6d8c234-2d8c-488b-9ea9-3969e9c430bf\" (UID: \"d6d8c234-2d8c-488b-9ea9-3969e9c430bf\") "
Oct 30 13:55:52.729540 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:52.729471    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-workload-certs\") pod \"d6d8c234-2d8c-488b-9ea9-3969e9c430bf\" (UID: \"d6d8c234-2d8c-488b-9ea9-3969e9c430bf\") "
Oct 30 13:55:52.729540 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:52.729520    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-workload-socket" pod "d6d8c234-2d8c-488b-9ea9-3969e9c430bf" (UID: "d6d8c234-2d8c-488b-9ea9-3969e9c430bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:52.729659 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:52.729596    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-istiod-ca-cert\") pod \"d6d8c234-2d8c-488b-9ea9-3969e9c430bf\" (UID: \"d6d8c234-2d8c-488b-9ea9-3969e9c430bf\") "
Oct 30 13:55:52.729760 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:52.729741    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-credential-socket" pod "d6d8c234-2d8c-488b-9ea9-3969e9c430bf" (UID: "d6d8c234-2d8c-488b-9ea9-3969e9c430bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:52.729967 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:52.729900    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-istio-data" pod "d6d8c234-2d8c-488b-9ea9-3969e9c430bf" (UID: "d6d8c234-2d8c-488b-9ea9-3969e9c430bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:52.729967 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:52.729938    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-workload-certs" pod "d6d8c234-2d8c-488b-9ea9-3969e9c430bf" (UID: "d6d8c234-2d8c-488b-9ea9-3969e9c430bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:52.730254 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:52.730223    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-istiod-ca-cert" pod "d6d8c234-2d8c-488b-9ea9-3969e9c430bf" (UID: "d6d8c234-2d8c-488b-9ea9-3969e9c430bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:55:52.730338 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:52.730289    5093 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-workload-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:55:52.730338 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:52.730307    5093 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-credential-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:55:52.730338 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:52.730316    5093 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-workload-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:55:52.730338 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:52.730327    5093 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-istio-data\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:55:52.731927 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:52.731880    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-istio-podinfo" pod "d6d8c234-2d8c-488b-9ea9-3969e9c430bf" (UID: "d6d8c234-2d8c-488b-9ea9-3969e9c430bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:55:52.732077 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:52.732054    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-istio-token" pod "d6d8c234-2d8c-488b-9ea9-3969e9c430bf" (UID: "d6d8c234-2d8c-488b-9ea9-3969e9c430bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:55:52.732359 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:52.732337    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-istio-envoy" pod "d6d8c234-2d8c-488b-9ea9-3969e9c430bf" (UID: "d6d8c234-2d8c-488b-9ea9-3969e9c430bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:52.732613 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:52.732592    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-kube-api-access-bsr6r" pod "d6d8c234-2d8c-488b-9ea9-3969e9c430bf" (UID: "d6d8c234-2d8c-488b-9ea9-3969e9c430bf"). InnerVolumeSpecName "kube-api-access-bsr6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:55:52.832777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:52.832680    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsr6r\" (UniqueName: \"kubernetes.io/projected/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-kube-api-access-bsr6r\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:55:52.832777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:52.832730    5093 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-istio-podinfo\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:55:52.832777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:52.832746    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:55:52.832777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:52.832758    5093 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-istio-envoy\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:55:52.832777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:52.832771    5093 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6d8c234-2d8c-488b-9ea9-3969e9c430bf-istiod-ca-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:55:52.844403 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:52.844366    5093 generic.go:356] "Generic (PLEG): container finished" podID="d6d8c234-2d8c-488b-9ea9-3969e9c430bf" containerID="87530e3ee9e3634a95376323c464547ab7e51a78d46c02c287a4c2af699a3d13" exitCode=0
Oct 30 13:55:52.844559 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:52.844431    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c"
Oct 30 13:55:52.844559 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:52.844447    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c" event={"ID":"d6d8c234-2d8c-488b-9ea9-3969e9c430bf","Type":"ContainerDied","Data":"87530e3ee9e3634a95376323c464547ab7e51a78d46c02c287a4c2af699a3d13"}
Oct 30 13:55:52.844559 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:52.844507    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c" event={"ID":"d6d8c234-2d8c-488b-9ea9-3969e9c430bf","Type":"ContainerDied","Data":"63d21876a1ecd60ac896052d9953bb702d97de68cff68905956eb33ddcd2dca8"}
Oct 30 13:55:52.844559 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:52.844531    5093 scope.go:122] "RemoveContainer" containerID="87530e3ee9e3634a95376323c464547ab7e51a78d46c02c287a4c2af699a3d13"
Oct 30 13:55:52.858295 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:52.858270    5093 scope.go:122] "RemoveContainer" containerID="87530e3ee9e3634a95376323c464547ab7e51a78d46c02c287a4c2af699a3d13"
Oct 30 13:55:52.858602 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:55:52.858576    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87530e3ee9e3634a95376323c464547ab7e51a78d46c02c287a4c2af699a3d13\": container with ID starting with 87530e3ee9e3634a95376323c464547ab7e51a78d46c02c287a4c2af699a3d13 not found: ID does not exist" containerID="87530e3ee9e3634a95376323c464547ab7e51a78d46c02c287a4c2af699a3d13"
Oct 30 13:55:52.858655 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:52.858619    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87530e3ee9e3634a95376323c464547ab7e51a78d46c02c287a4c2af699a3d13"} err="failed to get container status \"87530e3ee9e3634a95376323c464547ab7e51a78d46c02c287a4c2af699a3d13\": rpc error: code = NotFound desc = could not find container \"87530e3ee9e3634a95376323c464547ab7e51a78d46c02c287a4c2af699a3d13\": container with ID starting with 87530e3ee9e3634a95376323c464547ab7e51a78d46c02c287a4c2af699a3d13 not found: ID does not exist"
Oct 30 13:55:52.901587 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:52.901544    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c"]
Oct 30 13:55:52.906243 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:52.906207    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hbc8c"]
Oct 30 13:55:53.002451 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:55:53.002408    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6d8c234-2d8c-488b-9ea9-3969e9c430bf" path="/var/lib/kubelet/pods/d6d8c234-2d8c-488b-9ea9-3969e9c430bf/volumes"
Oct 30 13:56:21.170016 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:56:21.169962    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:56:51.169632 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:56:51.169582    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:57:21.175364 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:57:21.175322    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:57:28.689101 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:57:28.689061    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq"]
Oct 30 13:57:28.689744 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:57:28.689351    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq" podUID="a79305d1-bc2a-448f-be51-58affd67dd8a" containerName="istio-proxy" containerID="cri-o://f725306152f7cf8d7dc58e3d87719ab4eb14639642021784d881db300352b48a" gracePeriod=30
Oct 30 13:57:30.478819 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:57:30.478775    5093 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:57:30.479231 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:57:30.478879    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq" podUID="a79305d1-bc2a-448f-be51-58affd67dd8a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:57:34.049813 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:57:34.049782    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq"
Oct 30 13:57:34.140266 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:57:34.140214    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a79305d1-bc2a-448f-be51-58affd67dd8a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a79305d1-bc2a-448f-be51-58affd67dd8a-istiod-ca-cert\") pod \"a79305d1-bc2a-448f-be51-58affd67dd8a\" (UID: \"a79305d1-bc2a-448f-be51-58affd67dd8a\") "
Oct 30 13:57:34.140464 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:57:34.140384    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a79305d1-bc2a-448f-be51-58affd67dd8a-kube-api-access-7mhf8\" (UniqueName: \"kubernetes.io/projected/a79305d1-bc2a-448f-be51-58affd67dd8a-kube-api-access-7mhf8\") pod \"a79305d1-bc2a-448f-be51-58affd67dd8a\" (UID: \"a79305d1-bc2a-448f-be51-58affd67dd8a\") "
Oct 30 13:57:34.140464 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:57:34.140418    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a79305d1-bc2a-448f-be51-58affd67dd8a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a79305d1-bc2a-448f-be51-58affd67dd8a-istio-envoy\") pod \"a79305d1-bc2a-448f-be51-58affd67dd8a\" (UID: \"a79305d1-bc2a-448f-be51-58affd67dd8a\") "
Oct 30 13:57:34.140464 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:57:34.140458    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a79305d1-bc2a-448f-be51-58affd67dd8a-istio-token\" (UniqueName: \"kubernetes.io/projected/a79305d1-bc2a-448f-be51-58affd67dd8a-istio-token\") pod \"a79305d1-bc2a-448f-be51-58affd67dd8a\" (UID: \"a79305d1-bc2a-448f-be51-58affd67dd8a\") "
Oct 30 13:57:34.140663 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:57:34.140511    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a79305d1-bc2a-448f-be51-58affd67dd8a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79305d1-bc2a-448f-be51-58affd67dd8a-workload-socket\") pod \"a79305d1-bc2a-448f-be51-58affd67dd8a\" (UID: \"a79305d1-bc2a-448f-be51-58affd67dd8a\") "
Oct 30 13:57:34.140663 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:57:34.140537    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a79305d1-bc2a-448f-be51-58affd67dd8a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a79305d1-bc2a-448f-be51-58affd67dd8a-istio-podinfo\") pod \"a79305d1-bc2a-448f-be51-58affd67dd8a\" (UID: \"a79305d1-bc2a-448f-be51-58affd67dd8a\") "
Oct 30 13:57:34.140663 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:57:34.140614    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a79305d1-bc2a-448f-be51-58affd67dd8a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79305d1-bc2a-448f-be51-58affd67dd8a-credential-socket\") pod \"a79305d1-bc2a-448f-be51-58affd67dd8a\" (UID: \"a79305d1-bc2a-448f-be51-58affd67dd8a\") "
Oct 30 13:57:34.140836 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:57:34.140679    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a79305d1-bc2a-448f-be51-58affd67dd8a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a79305d1-bc2a-448f-be51-58affd67dd8a-workload-certs\") pod \"a79305d1-bc2a-448f-be51-58affd67dd8a\" (UID: \"a79305d1-bc2a-448f-be51-58affd67dd8a\") "
Oct 30 13:57:34.140836 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:57:34.140717    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a79305d1-bc2a-448f-be51-58affd67dd8a-istiod-ca-cert" pod "a79305d1-bc2a-448f-be51-58affd67dd8a" (UID: "a79305d1-bc2a-448f-be51-58affd67dd8a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:57:34.141017 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:57:34.140879    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a79305d1-bc2a-448f-be51-58affd67dd8a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a79305d1-bc2a-448f-be51-58affd67dd8a-istio-data\") pod \"a79305d1-bc2a-448f-be51-58affd67dd8a\" (UID: \"a79305d1-bc2a-448f-be51-58affd67dd8a\") "
Oct 30 13:57:34.141485 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:57:34.141456    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a79305d1-bc2a-448f-be51-58affd67dd8a-workload-certs" pod "a79305d1-bc2a-448f-be51-58affd67dd8a" (UID: "a79305d1-bc2a-448f-be51-58affd67dd8a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:34.141701 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:57:34.141629    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a79305d1-bc2a-448f-be51-58affd67dd8a-credential-socket" pod "a79305d1-bc2a-448f-be51-58affd67dd8a" (UID: "a79305d1-bc2a-448f-be51-58affd67dd8a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:34.141701 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:57:34.141692    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a79305d1-bc2a-448f-be51-58affd67dd8a-istio-data" pod "a79305d1-bc2a-448f-be51-58affd67dd8a" (UID: "a79305d1-bc2a-448f-be51-58affd67dd8a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:34.142893 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:57:34.141959    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a79305d1-bc2a-448f-be51-58affd67dd8a-workload-socket" pod "a79305d1-bc2a-448f-be51-58affd67dd8a" (UID: "a79305d1-bc2a-448f-be51-58affd67dd8a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:34.142893 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:57:34.142666    5093 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a79305d1-bc2a-448f-be51-58affd67dd8a-istiod-ca-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:57:34.142893 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:57:34.142770    5093 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79305d1-bc2a-448f-be51-58affd67dd8a-workload-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:57:34.142893 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:57:34.142786    5093 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79305d1-bc2a-448f-be51-58affd67dd8a-credential-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:57:34.142893 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:57:34.142800    5093 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a79305d1-bc2a-448f-be51-58affd67dd8a-workload-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:57:34.142893 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:57:34.142813    5093 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a79305d1-bc2a-448f-be51-58affd67dd8a-istio-data\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:57:34.149825 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:57:34.149793    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a79305d1-bc2a-448f-be51-58affd67dd8a-kube-api-access-7mhf8" pod "a79305d1-bc2a-448f-be51-58affd67dd8a" (UID: "a79305d1-bc2a-448f-be51-58affd67dd8a"). InnerVolumeSpecName "kube-api-access-7mhf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:57:34.151467 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:57:34.151436    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a79305d1-bc2a-448f-be51-58affd67dd8a-istio-token" pod "a79305d1-bc2a-448f-be51-58affd67dd8a" (UID: "a79305d1-bc2a-448f-be51-58affd67dd8a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:57:34.152020 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:57:34.151986    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a79305d1-bc2a-448f-be51-58affd67dd8a-istio-podinfo" pod "a79305d1-bc2a-448f-be51-58affd67dd8a" (UID: "a79305d1-bc2a-448f-be51-58affd67dd8a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:57:34.153293 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:57:34.153267    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a79305d1-bc2a-448f-be51-58affd67dd8a-istio-envoy" pod "a79305d1-bc2a-448f-be51-58affd67dd8a" (UID: "a79305d1-bc2a-448f-be51-58affd67dd8a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:34.244222 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:57:34.244177    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mhf8\" (UniqueName: \"kubernetes.io/projected/a79305d1-bc2a-448f-be51-58affd67dd8a-kube-api-access-7mhf8\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:57:34.244222 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:57:34.244212    5093 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a79305d1-bc2a-448f-be51-58affd67dd8a-istio-envoy\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:57:34.244222 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:57:34.244225    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a79305d1-bc2a-448f-be51-58affd67dd8a-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:57:34.244549 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:57:34.244237    5093 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a79305d1-bc2a-448f-be51-58affd67dd8a-istio-podinfo\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:57:34.848345 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:57:34.848304    5093 generic.go:356] "Generic (PLEG): container finished" podID="a79305d1-bc2a-448f-be51-58affd67dd8a" containerID="f725306152f7cf8d7dc58e3d87719ab4eb14639642021784d881db300352b48a" exitCode=0
Oct 30 13:57:34.848587 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:57:34.848362    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq"
Oct 30 13:57:34.848587 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:57:34.848433    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq" event={"ID":"a79305d1-bc2a-448f-be51-58affd67dd8a","Type":"ContainerDied","Data":"f725306152f7cf8d7dc58e3d87719ab4eb14639642021784d881db300352b48a"}
Oct 30 13:57:34.848587 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:57:34.848495    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq" event={"ID":"a79305d1-bc2a-448f-be51-58affd67dd8a","Type":"ContainerDied","Data":"6937527fef8f398531a125fc1347a5868dfbb7599748c609252ce3efd48275f3"}
Oct 30 13:57:34.848587 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:57:34.848525    5093 scope.go:122] "RemoveContainer" containerID="f725306152f7cf8d7dc58e3d87719ab4eb14639642021784d881db300352b48a"
Oct 30 13:57:34.863361 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:57:34.863333    5093 scope.go:122] "RemoveContainer" containerID="f725306152f7cf8d7dc58e3d87719ab4eb14639642021784d881db300352b48a"
Oct 30 13:57:34.863799 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:57:34.863769    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f725306152f7cf8d7dc58e3d87719ab4eb14639642021784d881db300352b48a\": container with ID starting with f725306152f7cf8d7dc58e3d87719ab4eb14639642021784d881db300352b48a not found: ID does not exist" containerID="f725306152f7cf8d7dc58e3d87719ab4eb14639642021784d881db300352b48a"
Oct 30 13:57:34.863938 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:57:34.863806    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f725306152f7cf8d7dc58e3d87719ab4eb14639642021784d881db300352b48a"} err="failed to get container status \"f725306152f7cf8d7dc58e3d87719ab4eb14639642021784d881db300352b48a\": rpc error: code = NotFound desc = could not find container \"f725306152f7cf8d7dc58e3d87719ab4eb14639642021784d881db300352b48a\": container with ID starting with f725306152f7cf8d7dc58e3d87719ab4eb14639642021784d881db300352b48a not found: ID does not exist"
Oct 30 13:57:34.922944 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:57:34.922909    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq"]
Oct 30 13:57:34.928797 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:57:34.928756    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-cjdzq"]
Oct 30 13:57:35.005642 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:57:35.005605    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a79305d1-bc2a-448f-be51-58affd67dd8a" path="/var/lib/kubelet/pods/a79305d1-bc2a-448f-be51-58affd67dd8a/volumes"
Oct 30 13:57:44.637336 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:57:44.636933    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-996b8b/test-model-996b8b-kserve-b594c475-hrhvr"]
Oct 30 13:57:51.174336 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:57:51.174286    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:57:59.670836 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:57:59.670779    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-996b8b/test-model-996b8b-kserve-b594c475-hrhvr" podUID="6e45d640-397e-420f-b746-c25bc3e96739" containerName="main" containerID="cri-o://0ad255d3f2e25b7c91b6503d695904c383e90412917bca23500be2dee8d4b68c" gracePeriod=45
Oct 30 13:57:59.679952 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:57:59.679904    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-996b8b/test-model-996b8b-kserve-b594c475-hrhvr" podUID="6e45d640-397e-420f-b746-c25bc3e96739" containerName="main" probeResult="failure" output="Get \"https://10.132.0.183:8000/ready\": EOF"
Oct 30 13:57:59.976964 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:57:59.976934    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-996b8b/test-model-996b8b-kserve-b594c475-hrhvr"
Oct 30 13:58:00.078499 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:00.078449    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e45d640-397e-420f-b746-c25bc3e96739-kube-api-access-d7s9c\" (UniqueName: \"kubernetes.io/projected/6e45d640-397e-420f-b746-c25bc3e96739-kube-api-access-d7s9c\") pod \"6e45d640-397e-420f-b746-c25bc3e96739\" (UID: \"6e45d640-397e-420f-b746-c25bc3e96739\") "
Oct 30 13:58:00.078769 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:00.078519    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e45d640-397e-420f-b746-c25bc3e96739-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e45d640-397e-420f-b746-c25bc3e96739-tmp-dir\") pod \"6e45d640-397e-420f-b746-c25bc3e96739\" (UID: \"6e45d640-397e-420f-b746-c25bc3e96739\") "
Oct 30 13:58:00.078769 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:00.078562    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e45d640-397e-420f-b746-c25bc3e96739-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e45d640-397e-420f-b746-c25bc3e96739-dshm\") pod \"6e45d640-397e-420f-b746-c25bc3e96739\" (UID: \"6e45d640-397e-420f-b746-c25bc3e96739\") "
Oct 30 13:58:00.078769 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:00.078592    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e45d640-397e-420f-b746-c25bc3e96739-tls-certs\" (UniqueName: \"kubernetes.io/secret/6e45d640-397e-420f-b746-c25bc3e96739-tls-certs\") pod \"6e45d640-397e-420f-b746-c25bc3e96739\" (UID: \"6e45d640-397e-420f-b746-c25bc3e96739\") "
Oct 30 13:58:00.078966 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:00.078808    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e45d640-397e-420f-b746-c25bc3e96739-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e45d640-397e-420f-b746-c25bc3e96739-model-cache\") pod \"6e45d640-397e-420f-b746-c25bc3e96739\" (UID: \"6e45d640-397e-420f-b746-c25bc3e96739\") "
Oct 30 13:58:00.078966 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:00.078861    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e45d640-397e-420f-b746-c25bc3e96739-home\" (UniqueName: \"kubernetes.io/empty-dir/6e45d640-397e-420f-b746-c25bc3e96739-home\") pod \"6e45d640-397e-420f-b746-c25bc3e96739\" (UID: \"6e45d640-397e-420f-b746-c25bc3e96739\") "
Oct 30 13:58:00.080002 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:00.079966    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e45d640-397e-420f-b746-c25bc3e96739-home" pod "6e45d640-397e-420f-b746-c25bc3e96739" (UID: "6e45d640-397e-420f-b746-c25bc3e96739"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:58:00.080182 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:00.080049    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e45d640-397e-420f-b746-c25bc3e96739-model-cache" pod "6e45d640-397e-420f-b746-c25bc3e96739" (UID: "6e45d640-397e-420f-b746-c25bc3e96739"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:58:00.080336 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:00.080311    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e45d640-397e-420f-b746-c25bc3e96739-tmp-dir" pod "6e45d640-397e-420f-b746-c25bc3e96739" (UID: "6e45d640-397e-420f-b746-c25bc3e96739"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:58:00.082067 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:00.082035    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e45d640-397e-420f-b746-c25bc3e96739-tls-certs" pod "6e45d640-397e-420f-b746-c25bc3e96739" (UID: "6e45d640-397e-420f-b746-c25bc3e96739"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:58:00.082776 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:00.082753    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e45d640-397e-420f-b746-c25bc3e96739-dshm" pod "6e45d640-397e-420f-b746-c25bc3e96739" (UID: "6e45d640-397e-420f-b746-c25bc3e96739"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:58:00.083027 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:00.083010    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e45d640-397e-420f-b746-c25bc3e96739-kube-api-access-d7s9c" pod "6e45d640-397e-420f-b746-c25bc3e96739" (UID: "6e45d640-397e-420f-b746-c25bc3e96739"). InnerVolumeSpecName "kube-api-access-d7s9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:58:00.098987 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:00.098950    5093 generic.go:356] "Generic (PLEG): container finished" podID="6e45d640-397e-420f-b746-c25bc3e96739" containerID="0ad255d3f2e25b7c91b6503d695904c383e90412917bca23500be2dee8d4b68c" exitCode=0
Oct 30 13:58:00.099148 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:00.099005    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-996b8b/test-model-996b8b-kserve-b594c475-hrhvr"
Oct 30 13:58:00.099300 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:00.099251    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-996b8b/test-model-996b8b-kserve-b594c475-hrhvr" event={"ID":"6e45d640-397e-420f-b746-c25bc3e96739","Type":"ContainerDied","Data":"0ad255d3f2e25b7c91b6503d695904c383e90412917bca23500be2dee8d4b68c"}
Oct 30 13:58:00.099440 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:00.099330    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-996b8b/test-model-996b8b-kserve-b594c475-hrhvr" event={"ID":"6e45d640-397e-420f-b746-c25bc3e96739","Type":"ContainerDied","Data":"5f8636809a04f4715fbffe38ba064ecc452f9cc2c3e50d379ba7169d1636a240"}
Oct 30 13:58:00.099440 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:00.099355    5093 scope.go:122] "RemoveContainer" containerID="0ad255d3f2e25b7c91b6503d695904c383e90412917bca23500be2dee8d4b68c"
Oct 30 13:58:00.138659 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:00.138633    5093 scope.go:122] "RemoveContainer" containerID="0ad255d3f2e25b7c91b6503d695904c383e90412917bca23500be2dee8d4b68c"
Oct 30 13:58:00.139010 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:58:00.138986    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ad255d3f2e25b7c91b6503d695904c383e90412917bca23500be2dee8d4b68c\": container with ID starting with 0ad255d3f2e25b7c91b6503d695904c383e90412917bca23500be2dee8d4b68c not found: ID does not exist" containerID="0ad255d3f2e25b7c91b6503d695904c383e90412917bca23500be2dee8d4b68c"
Oct 30 13:58:00.139099 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:00.139019    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ad255d3f2e25b7c91b6503d695904c383e90412917bca23500be2dee8d4b68c"} err="failed to get container status \"0ad255d3f2e25b7c91b6503d695904c383e90412917bca23500be2dee8d4b68c\": rpc error: code = NotFound desc = could not find container \"0ad255d3f2e25b7c91b6503d695904c383e90412917bca23500be2dee8d4b68c\": container with ID starting with 0ad255d3f2e25b7c91b6503d695904c383e90412917bca23500be2dee8d4b68c not found: ID does not exist"
Oct 30 13:58:00.170342 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:00.170304    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-996b8b/test-model-996b8b-kserve-b594c475-hrhvr"]
Oct 30 13:58:00.180833 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:00.180797    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-996b8b/test-model-996b8b-kserve-b594c475-hrhvr"]
Oct 30 13:58:00.181268 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:00.181238    5093 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e45d640-397e-420f-b746-c25bc3e96739-model-cache\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:58:00.181336 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:00.181275    5093 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e45d640-397e-420f-b746-c25bc3e96739-home\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:58:00.181336 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:00.181291    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7s9c\" (UniqueName: \"kubernetes.io/projected/6e45d640-397e-420f-b746-c25bc3e96739-kube-api-access-d7s9c\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:58:00.181336 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:00.181306    5093 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e45d640-397e-420f-b746-c25bc3e96739-tmp-dir\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:58:00.181336 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:00.181321    5093 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e45d640-397e-420f-b746-c25bc3e96739-dshm\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:58:00.181477 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:00.181334    5093 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e45d640-397e-420f-b746-c25bc3e96739-tls-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:58:01.003368 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:01.003330    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e45d640-397e-420f-b746-c25bc3e96739" path="/var/lib/kubelet/pods/6e45d640-397e-420f-b746-c25bc3e96739/volumes"
Oct 30 13:58:01.751273 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:01.751218    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jdj4_f93ee017-ab79-447c-90e0-5b85b71dba24/console-operator/1.log"
Oct 30 13:58:01.759764 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:01.759733    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tnm_3a675ac7-fbe1-449b-accf-bb0456ed0bd3/ovn-acl-logging/0.log"
Oct 30 13:58:01.766177 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:01.766144    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jdj4_f93ee017-ab79-447c-90e0-5b85b71dba24/console-operator/1.log"
Oct 30 13:58:01.773048 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:01.773024    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tnm_3a675ac7-fbe1-449b-accf-bb0456ed0bd3/ovn-acl-logging/0.log"
Oct 30 13:58:16.595557 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:16.595509    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-51605e/test-model-51605e-kserve-7d4cf4d78c-5527f"]
Oct 30 13:58:21.174416 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:58:21.174370    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:58:24.465572 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:24.465530    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-ca0243/test-model-ca0243-kserve-65cdfc7597-8b765"]
Oct 30 13:58:24.466420 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:24.466400    5093 cpu_manager.go:414] "containerMap: removing container" podUID="aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8" containerName="istio-proxy"
Oct 30 13:58:24.466420 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:24.466419    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8" containerName="istio-proxy"
Oct 30 13:58:24.466588 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:24.466439    5093 cpu_manager.go:414] "containerMap: removing container" podUID="6e45d640-397e-420f-b746-c25bc3e96739" containerName="main"
Oct 30 13:58:24.466588 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:24.466444    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e45d640-397e-420f-b746-c25bc3e96739" containerName="main"
Oct 30 13:58:24.466588 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:24.466454    5093 cpu_manager.go:414] "containerMap: removing container" podUID="a79305d1-bc2a-448f-be51-58affd67dd8a" containerName="istio-proxy"
Oct 30 13:58:24.466588 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:24.466461    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a79305d1-bc2a-448f-be51-58affd67dd8a" containerName="istio-proxy"
Oct 30 13:58:24.466588 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:24.466474    5093 cpu_manager.go:414] "containerMap: removing container" podUID="d6d8c234-2d8c-488b-9ea9-3969e9c430bf" containerName="istio-proxy"
Oct 30 13:58:24.466588 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:24.466479    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6d8c234-2d8c-488b-9ea9-3969e9c430bf" containerName="istio-proxy"
Oct 30 13:58:24.466588 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:24.466589    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e45d640-397e-420f-b746-c25bc3e96739" containerName="main"
Oct 30 13:58:24.466947 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:24.466600    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="a79305d1-bc2a-448f-be51-58affd67dd8a" containerName="istio-proxy"
Oct 30 13:58:24.466947 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:24.466609    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6d8c234-2d8c-488b-9ea9-3969e9c430bf" containerName="istio-proxy"
Oct 30 13:58:24.466947 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:24.466623    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaefd92b-7e59-4d40-9b4f-d6b6a21d97e8" containerName="istio-proxy"
Oct 30 13:58:24.472043 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:24.472019    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ca0243/test-model-ca0243-kserve-65cdfc7597-8b765"
Oct 30 13:58:24.492798 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:24.492767    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ca0243/test-model-ca0243-kserve-65cdfc7597-8b765"]
Oct 30 13:58:24.500432 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:24.500399    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ca0243\"/\"test-model-ca0243-kserve-self-signed-certs\""
Oct 30 13:58:24.500758 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:24.500727    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ca0243\"/\"default-dockercfg-42vt4\""
Oct 30 13:58:24.502184 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:24.502118    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ca0243\"/\"openshift-service-ca.crt\""
Oct 30 13:58:24.502342 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:24.502235    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ca0243\"/\"kube-root-ca.crt\""
Oct 30 13:58:24.589706 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:24.589668    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec8e38bf-56e5-4699-9814-1d17423be437-tmp-dir\") pod \"test-model-ca0243-kserve-65cdfc7597-8b765\" (UID: \"ec8e38bf-56e5-4699-9814-1d17423be437\") " pod="ai-tenant-e2e-inf-b-ca0243/test-model-ca0243-kserve-65cdfc7597-8b765"
Oct 30 13:58:24.589911 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:24.589725    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec8e38bf-56e5-4699-9814-1d17423be437-home\") pod \"test-model-ca0243-kserve-65cdfc7597-8b765\" (UID: \"ec8e38bf-56e5-4699-9814-1d17423be437\") " pod="ai-tenant-e2e-inf-b-ca0243/test-model-ca0243-kserve-65cdfc7597-8b765"
Oct 30 13:58:24.589911 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:24.589877    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtp9r\" (UniqueName: \"kubernetes.io/projected/ec8e38bf-56e5-4699-9814-1d17423be437-kube-api-access-vtp9r\") pod \"test-model-ca0243-kserve-65cdfc7597-8b765\" (UID: \"ec8e38bf-56e5-4699-9814-1d17423be437\") " pod="ai-tenant-e2e-inf-b-ca0243/test-model-ca0243-kserve-65cdfc7597-8b765"
Oct 30 13:58:24.590006 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:24.589987    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec8e38bf-56e5-4699-9814-1d17423be437-dshm\") pod \"test-model-ca0243-kserve-65cdfc7597-8b765\" (UID: \"ec8e38bf-56e5-4699-9814-1d17423be437\") " pod="ai-tenant-e2e-inf-b-ca0243/test-model-ca0243-kserve-65cdfc7597-8b765"
Oct 30 13:58:24.590171 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:24.590151    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec8e38bf-56e5-4699-9814-1d17423be437-model-cache\") pod \"test-model-ca0243-kserve-65cdfc7597-8b765\" (UID: \"ec8e38bf-56e5-4699-9814-1d17423be437\") " pod="ai-tenant-e2e-inf-b-ca0243/test-model-ca0243-kserve-65cdfc7597-8b765"
Oct 30 13:58:24.590225 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:24.590180    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec8e38bf-56e5-4699-9814-1d17423be437-tls-certs\") pod \"test-model-ca0243-kserve-65cdfc7597-8b765\" (UID: \"ec8e38bf-56e5-4699-9814-1d17423be437\") " pod="ai-tenant-e2e-inf-b-ca0243/test-model-ca0243-kserve-65cdfc7597-8b765"
Oct 30 13:58:24.697156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:24.694012    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec8e38bf-56e5-4699-9814-1d17423be437-home\") pod \"test-model-ca0243-kserve-65cdfc7597-8b765\" (UID: \"ec8e38bf-56e5-4699-9814-1d17423be437\") " pod="ai-tenant-e2e-inf-b-ca0243/test-model-ca0243-kserve-65cdfc7597-8b765"
Oct 30 13:58:24.697156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:24.694236    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtp9r\" (UniqueName: \"kubernetes.io/projected/ec8e38bf-56e5-4699-9814-1d17423be437-kube-api-access-vtp9r\") pod \"test-model-ca0243-kserve-65cdfc7597-8b765\" (UID: \"ec8e38bf-56e5-4699-9814-1d17423be437\") " pod="ai-tenant-e2e-inf-b-ca0243/test-model-ca0243-kserve-65cdfc7597-8b765"
Oct 30 13:58:24.697156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:24.694339    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec8e38bf-56e5-4699-9814-1d17423be437-dshm\") pod \"test-model-ca0243-kserve-65cdfc7597-8b765\" (UID: \"ec8e38bf-56e5-4699-9814-1d17423be437\") " pod="ai-tenant-e2e-inf-b-ca0243/test-model-ca0243-kserve-65cdfc7597-8b765"
Oct 30 13:58:24.697156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:24.694504    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec8e38bf-56e5-4699-9814-1d17423be437-model-cache\") pod \"test-model-ca0243-kserve-65cdfc7597-8b765\" (UID: \"ec8e38bf-56e5-4699-9814-1d17423be437\") " pod="ai-tenant-e2e-inf-b-ca0243/test-model-ca0243-kserve-65cdfc7597-8b765"
Oct 30 13:58:24.697156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:24.694532    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec8e38bf-56e5-4699-9814-1d17423be437-tls-certs\") pod \"test-model-ca0243-kserve-65cdfc7597-8b765\" (UID: \"ec8e38bf-56e5-4699-9814-1d17423be437\") " pod="ai-tenant-e2e-inf-b-ca0243/test-model-ca0243-kserve-65cdfc7597-8b765"
Oct 30 13:58:24.697156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:24.694687    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec8e38bf-56e5-4699-9814-1d17423be437-tmp-dir\") pod \"test-model-ca0243-kserve-65cdfc7597-8b765\" (UID: \"ec8e38bf-56e5-4699-9814-1d17423be437\") " pod="ai-tenant-e2e-inf-b-ca0243/test-model-ca0243-kserve-65cdfc7597-8b765"
Oct 30 13:58:24.697156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:24.695532    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec8e38bf-56e5-4699-9814-1d17423be437-home\") pod \"test-model-ca0243-kserve-65cdfc7597-8b765\" (UID: \"ec8e38bf-56e5-4699-9814-1d17423be437\") " pod="ai-tenant-e2e-inf-b-ca0243/test-model-ca0243-kserve-65cdfc7597-8b765"
Oct 30 13:58:24.697156 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:24.695798    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec8e38bf-56e5-4699-9814-1d17423be437-model-cache\") pod \"test-model-ca0243-kserve-65cdfc7597-8b765\" (UID: \"ec8e38bf-56e5-4699-9814-1d17423be437\") " pod="ai-tenant-e2e-inf-b-ca0243/test-model-ca0243-kserve-65cdfc7597-8b765"
Oct 30 13:58:24.728216 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:24.718308    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec8e38bf-56e5-4699-9814-1d17423be437-tmp-dir\") pod \"test-model-ca0243-kserve-65cdfc7597-8b765\" (UID: \"ec8e38bf-56e5-4699-9814-1d17423be437\") " pod="ai-tenant-e2e-inf-b-ca0243/test-model-ca0243-kserve-65cdfc7597-8b765"
Oct 30 13:58:24.735985 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:24.735907    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec8e38bf-56e5-4699-9814-1d17423be437-tls-certs\") pod \"test-model-ca0243-kserve-65cdfc7597-8b765\" (UID: \"ec8e38bf-56e5-4699-9814-1d17423be437\") " pod="ai-tenant-e2e-inf-b-ca0243/test-model-ca0243-kserve-65cdfc7597-8b765"
Oct 30 13:58:24.751084 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:24.741655    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec8e38bf-56e5-4699-9814-1d17423be437-dshm\") pod \"test-model-ca0243-kserve-65cdfc7597-8b765\" (UID: \"ec8e38bf-56e5-4699-9814-1d17423be437\") " pod="ai-tenant-e2e-inf-b-ca0243/test-model-ca0243-kserve-65cdfc7597-8b765"
Oct 30 13:58:24.757982 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:24.757911    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtp9r\" (UniqueName: \"kubernetes.io/projected/ec8e38bf-56e5-4699-9814-1d17423be437-kube-api-access-vtp9r\") pod \"test-model-ca0243-kserve-65cdfc7597-8b765\" (UID: \"ec8e38bf-56e5-4699-9814-1d17423be437\") " pod="ai-tenant-e2e-inf-b-ca0243/test-model-ca0243-kserve-65cdfc7597-8b765"
Oct 30 13:58:24.786367 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:24.786333    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ca0243/test-model-ca0243-kserve-65cdfc7597-8b765"
Oct 30 13:58:25.109777 ip-10-0-134-31 kubenswrapper[5093]: W1030 13:58:25.109736    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec8e38bf_56e5_4699_9814_1d17423be437.slice/crio-9dd69aeda3983b972c7fb573ca91b229c605d67d47dc472aff9ea685aa75def8 WatchSource:0}: Error finding container 9dd69aeda3983b972c7fb573ca91b229c605d67d47dc472aff9ea685aa75def8: Status 404 returned error can't find the container with id 9dd69aeda3983b972c7fb573ca91b229c605d67d47dc472aff9ea685aa75def8
Oct 30 13:58:25.110209 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:25.110180    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ca0243/test-model-ca0243-kserve-65cdfc7597-8b765"]
Oct 30 13:58:25.353529 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:25.353480    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ca0243/test-model-ca0243-kserve-65cdfc7597-8b765" event={"ID":"ec8e38bf-56e5-4699-9814-1d17423be437","Type":"ContainerStarted","Data":"1070597f644c24611e6e3eaffb2e4e0617612aa7bffaadf584dfa38f9607a00a"}
Oct 30 13:58:25.353529 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:25.353527    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ca0243/test-model-ca0243-kserve-65cdfc7597-8b765" event={"ID":"ec8e38bf-56e5-4699-9814-1d17423be437","Type":"ContainerStarted","Data":"9dd69aeda3983b972c7fb573ca91b229c605d67d47dc472aff9ea685aa75def8"}
Oct 30 13:58:25.396857 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:25.396751    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-ca0243/test-model-ca0243-kserve-65cdfc7597-8b765" podStartSLOduration=1.396736239 podStartE2EDuration="1.396736239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:58:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:58:25.394139816 +0000 UTC m=+3324.541407058" watchObservedRunningTime="2026-10-30 13:58:25.396736239 +0000 UTC m=+3324.544003479"
Oct 30 13:58:25.787059 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:25.787021    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ca0243/test-model-ca0243-kserve-65cdfc7597-8b765"
Oct 30 13:58:31.619225 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:31.619178    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-51605e/test-model-51605e-kserve-7d4cf4d78c-5527f" podUID="14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5" containerName="main" containerID="cri-o://09963cb8c2e410658cbdd0cb2f4663702174a43f3370a92cbbd37a915bd87e23" gracePeriod=45
Oct 30 13:58:31.722309 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:31.722264    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-51605e/test-model-51605e-kserve-7d4cf4d78c-5527f" podUID="14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.184:8000/ready\": dial tcp 10.132.0.184:8000: connect: connection refused"
Oct 30 13:58:31.942612 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:31.942579    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-51605e/test-model-51605e-kserve-7d4cf4d78c-5527f"
Oct 30 13:58:32.121191 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:32.121120    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5-tls-certs\" (UniqueName: \"kubernetes.io/secret/14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5-tls-certs\") pod \"14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5\" (UID: \"14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5\") "
Oct 30 13:58:32.121371 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:32.121327    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5-tmp-dir\") pod \"14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5\" (UID: \"14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5\") "
Oct 30 13:58:32.121371 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:32.121357    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5-model-cache\") pod \"14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5\" (UID: \"14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5\") "
Oct 30 13:58:32.121454 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:32.121432    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5-kube-api-access-w69cx\" (UniqueName: \"kubernetes.io/projected/14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5-kube-api-access-w69cx\") pod \"14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5\" (UID: \"14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5\") "
Oct 30 13:58:32.121494 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:32.121472    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5-dshm\") pod \"14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5\" (UID: \"14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5\") "
Oct 30 13:58:32.121549 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:32.121511    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5-home\" (UniqueName: \"kubernetes.io/empty-dir/14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5-home\") pod \"14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5\" (UID: \"14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5\") "
Oct 30 13:58:32.121631 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:32.121606    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5-tmp-dir" pod "14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5" (UID: "14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:58:32.121757 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:32.121711    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5-model-cache" pod "14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5" (UID: "14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:58:32.121862 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:32.121810    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5-home" pod "14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5" (UID: "14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:58:32.122407 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:32.122377    5093 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5-home\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:58:32.122539 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:32.122411    5093 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5-tmp-dir\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:58:32.122539 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:32.122428    5093 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5-model-cache\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:58:32.123846 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:32.123817    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5-dshm" pod "14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5" (UID: "14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:58:32.123944 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:32.123860    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5-tls-certs" pod "14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5" (UID: "14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:58:32.123944 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:32.123905    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5-kube-api-access-w69cx" pod "14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5" (UID: "14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5"). InnerVolumeSpecName "kube-api-access-w69cx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:58:32.223765 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:32.223733    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w69cx\" (UniqueName: \"kubernetes.io/projected/14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5-kube-api-access-w69cx\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:58:32.223765 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:32.223761    5093 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5-dshm\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:58:32.223765 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:32.223771    5093 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5-tls-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:58:32.413856 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:32.413818    5093 generic.go:356] "Generic (PLEG): container finished" podID="14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5" containerID="09963cb8c2e410658cbdd0cb2f4663702174a43f3370a92cbbd37a915bd87e23" exitCode=0
Oct 30 13:58:32.414034 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:32.413866    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-51605e/test-model-51605e-kserve-7d4cf4d78c-5527f"
Oct 30 13:58:32.414034 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:32.413892    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-51605e/test-model-51605e-kserve-7d4cf4d78c-5527f" event={"ID":"14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5","Type":"ContainerDied","Data":"09963cb8c2e410658cbdd0cb2f4663702174a43f3370a92cbbd37a915bd87e23"}
Oct 30 13:58:32.414034 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:32.413943    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-51605e/test-model-51605e-kserve-7d4cf4d78c-5527f" event={"ID":"14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5","Type":"ContainerDied","Data":"fdf4a7c05f86010b7dc6c3a98dba91874109785de72c766ee4e17be7ae382382"}
Oct 30 13:58:32.414034 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:32.413963    5093 scope.go:122] "RemoveContainer" containerID="09963cb8c2e410658cbdd0cb2f4663702174a43f3370a92cbbd37a915bd87e23"
Oct 30 13:58:32.427326 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:32.427293    5093 scope.go:122] "RemoveContainer" containerID="09963cb8c2e410658cbdd0cb2f4663702174a43f3370a92cbbd37a915bd87e23"
Oct 30 13:58:32.427626 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:58:32.427591    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09963cb8c2e410658cbdd0cb2f4663702174a43f3370a92cbbd37a915bd87e23\": container with ID starting with 09963cb8c2e410658cbdd0cb2f4663702174a43f3370a92cbbd37a915bd87e23 not found: ID does not exist" containerID="09963cb8c2e410658cbdd0cb2f4663702174a43f3370a92cbbd37a915bd87e23"
Oct 30 13:58:32.427740 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:32.427635    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09963cb8c2e410658cbdd0cb2f4663702174a43f3370a92cbbd37a915bd87e23"} err="failed to get container status \"09963cb8c2e410658cbdd0cb2f4663702174a43f3370a92cbbd37a915bd87e23\": rpc error: code = NotFound desc = could not find container \"09963cb8c2e410658cbdd0cb2f4663702174a43f3370a92cbbd37a915bd87e23\": container with ID starting with 09963cb8c2e410658cbdd0cb2f4663702174a43f3370a92cbbd37a915bd87e23 not found: ID does not exist"
Oct 30 13:58:32.471504 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:32.471473    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-51605e/test-model-51605e-kserve-7d4cf4d78c-5527f"]
Oct 30 13:58:32.483831 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:32.483737    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-51605e/test-model-51605e-kserve-7d4cf4d78c-5527f"]
Oct 30 13:58:33.001368 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:33.001332    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5" path="/var/lib/kubelet/pods/14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5/volumes"
Oct 30 13:58:34.787191 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:34.787156    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-ca0243/test-model-ca0243-kserve-65cdfc7597-8b765"
Oct 30 13:58:34.800264 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:34.800230    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-ca0243/test-model-ca0243-kserve-65cdfc7597-8b765"
Oct 30 13:58:35.257195 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:35.257152    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x"]
Oct 30 13:58:35.257407 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:35.257382    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x" podUID="1a9a0d53-3fb0-414a-a543-f5dd4ed39eef" containerName="istio-proxy" containerID="cri-o://490b4fae81b4f157e902505c0a49e4e7eadb9bd26ef722a0a577ca88ccbc80a5" gracePeriod=30
Oct 30 13:58:35.475953 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:35.475924    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ca0243/test-model-ca0243-kserve-65cdfc7597-8b765"
Oct 30 13:58:40.079541 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:40.079503    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5494c9-684fdf5d4b-dp78d"]
Oct 30 13:58:40.080053 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:40.079725    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5494c9-684fdf5d4b-dp78d" podUID="8aaca537-18ae-4b3a-9da2-3c834d740100" containerName="maas-api" containerID="cri-o://0d7423d7719916d1b981a5f0ad33dd1a3cd7d7534dd9abc3529c24031720751a" gracePeriod=30
Oct 30 13:58:40.452651 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:40.452616    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5494c9-684fdf5d4b-dp78d"
Oct 30 13:58:40.530545 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:40.530014    5093 generic.go:356] "Generic (PLEG): container finished" podID="8aaca537-18ae-4b3a-9da2-3c834d740100" containerID="0d7423d7719916d1b981a5f0ad33dd1a3cd7d7534dd9abc3529c24031720751a" exitCode=0
Oct 30 13:58:40.530545 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:40.530179    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5494c9-684fdf5d4b-dp78d" event={"ID":"8aaca537-18ae-4b3a-9da2-3c834d740100","Type":"ContainerDied","Data":"0d7423d7719916d1b981a5f0ad33dd1a3cd7d7534dd9abc3529c24031720751a"}
Oct 30 13:58:40.530545 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:40.530242    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5494c9-684fdf5d4b-dp78d" event={"ID":"8aaca537-18ae-4b3a-9da2-3c834d740100","Type":"ContainerDied","Data":"8c32921d0fd9a73f765855b7b84bfc66efc9811e6f96c883df6edb8915a5f212"}
Oct 30 13:58:40.530545 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:40.530274    5093 scope.go:122] "RemoveContainer" containerID="0d7423d7719916d1b981a5f0ad33dd1a3cd7d7534dd9abc3529c24031720751a"
Oct 30 13:58:40.530545 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:40.530452    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5494c9-684fdf5d4b-dp78d"
Oct 30 13:58:40.594794 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:40.594026    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aaca537-18ae-4b3a-9da2-3c834d740100-kube-api-access-tlhbq\" (UniqueName: \"kubernetes.io/projected/8aaca537-18ae-4b3a-9da2-3c834d740100-kube-api-access-tlhbq\") pod \"8aaca537-18ae-4b3a-9da2-3c834d740100\" (UID: \"8aaca537-18ae-4b3a-9da2-3c834d740100\") "
Oct 30 13:58:40.596339 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:40.594934    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8aaca537-18ae-4b3a-9da2-3c834d740100-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8aaca537-18ae-4b3a-9da2-3c834d740100-maas-api-tls\") pod \"8aaca537-18ae-4b3a-9da2-3c834d740100\" (UID: \"8aaca537-18ae-4b3a-9da2-3c834d740100\") "
Oct 30 13:58:40.596339 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:40.595253    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8aaca537-18ae-4b3a-9da2-3c834d740100-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aaca537-18ae-4b3a-9da2-3c834d740100-openshift-service-ca\") pod \"8aaca537-18ae-4b3a-9da2-3c834d740100\" (UID: \"8aaca537-18ae-4b3a-9da2-3c834d740100\") "
Oct 30 13:58:40.596339 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:40.595306    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8aaca537-18ae-4b3a-9da2-3c834d740100-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8aaca537-18ae-4b3a-9da2-3c834d740100-metrics-certs\") pod \"8aaca537-18ae-4b3a-9da2-3c834d740100\" (UID: \"8aaca537-18ae-4b3a-9da2-3c834d740100\") "
Oct 30 13:58:40.621822 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:40.604212    5093 generic.go:356] "Generic (PLEG): container finished" podID="1a9a0d53-3fb0-414a-a543-f5dd4ed39eef" containerID="490b4fae81b4f157e902505c0a49e4e7eadb9bd26ef722a0a577ca88ccbc80a5" exitCode=0
Oct 30 13:58:40.621822 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:40.604571    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x" event={"ID":"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef","Type":"ContainerDied","Data":"490b4fae81b4f157e902505c0a49e4e7eadb9bd26ef722a0a577ca88ccbc80a5"}
Oct 30 13:58:40.621822 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:40.621755    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aaca537-18ae-4b3a-9da2-3c834d740100-openshift-service-ca" pod "8aaca537-18ae-4b3a-9da2-3c834d740100" (UID: "8aaca537-18ae-4b3a-9da2-3c834d740100"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:58:40.671093 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:40.667565    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aaca537-18ae-4b3a-9da2-3c834d740100-maas-api-tls" pod "8aaca537-18ae-4b3a-9da2-3c834d740100" (UID: "8aaca537-18ae-4b3a-9da2-3c834d740100"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:58:40.671093 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:40.668242    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aaca537-18ae-4b3a-9da2-3c834d740100-metrics-certs" pod "8aaca537-18ae-4b3a-9da2-3c834d740100" (UID: "8aaca537-18ae-4b3a-9da2-3c834d740100"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:58:40.671093 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:40.668400    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aaca537-18ae-4b3a-9da2-3c834d740100-kube-api-access-tlhbq" pod "8aaca537-18ae-4b3a-9da2-3c834d740100" (UID: "8aaca537-18ae-4b3a-9da2-3c834d740100"). InnerVolumeSpecName "kube-api-access-tlhbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:58:40.699269 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:40.699178    5093 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8aaca537-18ae-4b3a-9da2-3c834d740100-metrics-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:58:40.699269 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:40.699215    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlhbq\" (UniqueName: \"kubernetes.io/projected/8aaca537-18ae-4b3a-9da2-3c834d740100-kube-api-access-tlhbq\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:58:40.699269 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:40.699230    5093 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8aaca537-18ae-4b3a-9da2-3c834d740100-maas-api-tls\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:58:40.699269 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:40.699243    5093 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aaca537-18ae-4b3a-9da2-3c834d740100-openshift-service-ca\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:58:40.735653 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:40.735510    5093 scope.go:122] "RemoveContainer" containerID="0d7423d7719916d1b981a5f0ad33dd1a3cd7d7534dd9abc3529c24031720751a"
Oct 30 13:58:40.738369 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:58:40.736763    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d7423d7719916d1b981a5f0ad33dd1a3cd7d7534dd9abc3529c24031720751a\": container with ID starting with 0d7423d7719916d1b981a5f0ad33dd1a3cd7d7534dd9abc3529c24031720751a not found: ID does not exist" containerID="0d7423d7719916d1b981a5f0ad33dd1a3cd7d7534dd9abc3529c24031720751a"
Oct 30 13:58:40.738369 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:40.736810    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d7423d7719916d1b981a5f0ad33dd1a3cd7d7534dd9abc3529c24031720751a"} err="failed to get container status \"0d7423d7719916d1b981a5f0ad33dd1a3cd7d7534dd9abc3529c24031720751a\": rpc error: code = NotFound desc = could not find container \"0d7423d7719916d1b981a5f0ad33dd1a3cd7d7534dd9abc3529c24031720751a\": container with ID starting with 0d7423d7719916d1b981a5f0ad33dd1a3cd7d7534dd9abc3529c24031720751a not found: ID does not exist"
Oct 30 13:58:40.738369 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:40.736837    5093 scope.go:122] "RemoveContainer" containerID="e6d77701831809fd79f318075f38754db29a881178681c1154bc41ee91ddc86f"
Oct 30 13:58:40.758427 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:40.758405    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x"
Oct 30 13:58:40.882727 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:40.882689    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5494c9-684fdf5d4b-dp78d"]
Oct 30 13:58:40.887160 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:40.887091    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5494c9-684fdf5d4b-dp78d"]
Oct 30 13:58:40.902340 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:40.902308    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-istio-data\") pod \"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef\" (UID: \"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef\") "
Oct 30 13:58:40.902504 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:40.902345    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-credential-socket\") pod \"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef\" (UID: \"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef\") "
Oct 30 13:58:40.902504 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:40.902461    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-workload-certs\") pod \"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef\" (UID: \"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef\") "
Oct 30 13:58:40.902504 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:40.902484    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-istio-podinfo\") pod \"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef\" (UID: \"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef\") "
Oct 30 13:58:40.902689 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:40.902514    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-workload-socket\") pod \"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef\" (UID: \"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef\") "
Oct 30 13:58:40.902689 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:40.902569    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-istio-envoy\") pod \"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef\" (UID: \"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef\") "
Oct 30 13:58:40.902689 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:40.902614    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-istiod-ca-cert\") pod \"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef\" (UID: \"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef\") "
Oct 30 13:58:40.902689 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:40.902607    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-istio-data" pod "1a9a0d53-3fb0-414a-a543-f5dd4ed39eef" (UID: "1a9a0d53-3fb0-414a-a543-f5dd4ed39eef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:58:40.902689 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:40.902658    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-kube-api-access-jzv4f\" (UniqueName: \"kubernetes.io/projected/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-kube-api-access-jzv4f\") pod \"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef\" (UID: \"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef\") "
Oct 30 13:58:40.902912 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:40.902689    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-credential-socket" pod "1a9a0d53-3fb0-414a-a543-f5dd4ed39eef" (UID: "1a9a0d53-3fb0-414a-a543-f5dd4ed39eef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:58:40.902912 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:40.902784    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-istio-token\" (UniqueName: \"kubernetes.io/projected/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-istio-token\") pod \"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef\" (UID: \"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef\") "
Oct 30 13:58:40.902912 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:40.902784    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-workload-certs" pod "1a9a0d53-3fb0-414a-a543-f5dd4ed39eef" (UID: "1a9a0d53-3fb0-414a-a543-f5dd4ed39eef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:58:40.903068 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:40.902912    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-workload-socket" pod "1a9a0d53-3fb0-414a-a543-f5dd4ed39eef" (UID: "1a9a0d53-3fb0-414a-a543-f5dd4ed39eef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:58:40.903339 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:40.903315    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-istiod-ca-cert" pod "1a9a0d53-3fb0-414a-a543-f5dd4ed39eef" (UID: "1a9a0d53-3fb0-414a-a543-f5dd4ed39eef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:58:40.903469 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:40.903372    5093 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-workload-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:58:40.903469 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:40.903392    5093 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-workload-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:58:40.903469 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:40.903404    5093 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-istio-data\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:58:40.903469 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:40.903420    5093 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-credential-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:58:40.905439 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:40.905414    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-kube-api-access-jzv4f" pod "1a9a0d53-3fb0-414a-a543-f5dd4ed39eef" (UID: "1a9a0d53-3fb0-414a-a543-f5dd4ed39eef"). InnerVolumeSpecName "kube-api-access-jzv4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:58:40.905530 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:40.905440    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-istio-envoy" pod "1a9a0d53-3fb0-414a-a543-f5dd4ed39eef" (UID: "1a9a0d53-3fb0-414a-a543-f5dd4ed39eef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:58:40.905530 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:40.905446    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-istio-podinfo" pod "1a9a0d53-3fb0-414a-a543-f5dd4ed39eef" (UID: "1a9a0d53-3fb0-414a-a543-f5dd4ed39eef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:58:40.905530 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:40.905466    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-istio-token" pod "1a9a0d53-3fb0-414a-a543-f5dd4ed39eef" (UID: "1a9a0d53-3fb0-414a-a543-f5dd4ed39eef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:58:41.003209 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:41.003171    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8aaca537-18ae-4b3a-9da2-3c834d740100" path="/var/lib/kubelet/pods/8aaca537-18ae-4b3a-9da2-3c834d740100/volumes"
Oct 30 13:58:41.004485 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:41.004463    5093 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-istio-podinfo\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:58:41.004566 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:41.004487    5093 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-istio-envoy\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:58:41.004566 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:41.004496    5093 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-istiod-ca-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:58:41.004566 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:41.004504    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzv4f\" (UniqueName: \"kubernetes.io/projected/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-kube-api-access-jzv4f\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:58:41.004566 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:41.004512    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:58:41.616903 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:41.616872    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x"
Oct 30 13:58:41.616903 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:41.616896    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x" event={"ID":"1a9a0d53-3fb0-414a-a543-f5dd4ed39eef","Type":"ContainerDied","Data":"a557d49240ef6dd7446c711b8d909c36223a48397f601e9a9cb7f94574908c43"}
Oct 30 13:58:41.617492 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:41.616950    5093 scope.go:122] "RemoveContainer" containerID="490b4fae81b4f157e902505c0a49e4e7eadb9bd26ef722a0a577ca88ccbc80a5"
Oct 30 13:58:41.704390 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:41.704345    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x"]
Oct 30 13:58:41.709182 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:41.709151    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-wbr7x"]
Oct 30 13:58:43.001025 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:43.000976    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a9a0d53-3fb0-414a-a543-f5dd4ed39eef" path="/var/lib/kubelet/pods/1a9a0d53-3fb0-414a-a543-f5dd4ed39eef/volumes"
Oct 30 13:58:48.894190 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:48.894148    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx"]
Oct 30 13:58:48.894735 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:48.894446    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx" podUID="d474316d-631a-4b79-baaf-2e19fb16a211" containerName="istio-proxy" containerID="cri-o://b979fd59d3b2a04520ca52b9c57af7f10db30c394df8ba2a4d993b8fdab4e8d3" gracePeriod=30
Oct 30 13:58:49.452756 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:49.452715    5093 patch_prober.go:29] interesting pod/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:58:49.452947 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:49.452782    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx" podUID="d474316d-631a-4b79-baaf-2e19fb16a211" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:58:51.167448 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:58:51.167410    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:58:54.206017 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:54.205987    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx"
Oct 30 13:58:54.329766 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:54.329698    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d474316d-631a-4b79-baaf-2e19fb16a211-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d474316d-631a-4b79-baaf-2e19fb16a211-workload-socket\") pod \"d474316d-631a-4b79-baaf-2e19fb16a211\" (UID: \"d474316d-631a-4b79-baaf-2e19fb16a211\") "
Oct 30 13:58:54.329766 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:54.329765    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d474316d-631a-4b79-baaf-2e19fb16a211-istio-token\" (UniqueName: \"kubernetes.io/projected/d474316d-631a-4b79-baaf-2e19fb16a211-istio-token\") pod \"d474316d-631a-4b79-baaf-2e19fb16a211\" (UID: \"d474316d-631a-4b79-baaf-2e19fb16a211\") "
Oct 30 13:58:54.330035 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:54.329811    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d474316d-631a-4b79-baaf-2e19fb16a211-kube-api-access-skms5\" (UniqueName: \"kubernetes.io/projected/d474316d-631a-4b79-baaf-2e19fb16a211-kube-api-access-skms5\") pod \"d474316d-631a-4b79-baaf-2e19fb16a211\" (UID: \"d474316d-631a-4b79-baaf-2e19fb16a211\") "
Oct 30 13:58:54.330035 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:54.329996    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d474316d-631a-4b79-baaf-2e19fb16a211-workload-socket" pod "d474316d-631a-4b79-baaf-2e19fb16a211" (UID: "d474316d-631a-4b79-baaf-2e19fb16a211"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:58:54.330187 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:54.330145    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d474316d-631a-4b79-baaf-2e19fb16a211-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d474316d-631a-4b79-baaf-2e19fb16a211-istio-data\") pod \"d474316d-631a-4b79-baaf-2e19fb16a211\" (UID: \"d474316d-631a-4b79-baaf-2e19fb16a211\") "
Oct 30 13:58:54.330242 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:54.330213    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d474316d-631a-4b79-baaf-2e19fb16a211-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d474316d-631a-4b79-baaf-2e19fb16a211-istio-envoy\") pod \"d474316d-631a-4b79-baaf-2e19fb16a211\" (UID: \"d474316d-631a-4b79-baaf-2e19fb16a211\") "
Oct 30 13:58:54.330311 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:54.330292    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d474316d-631a-4b79-baaf-2e19fb16a211-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d474316d-631a-4b79-baaf-2e19fb16a211-credential-socket\") pod \"d474316d-631a-4b79-baaf-2e19fb16a211\" (UID: \"d474316d-631a-4b79-baaf-2e19fb16a211\") "
Oct 30 13:58:54.330449 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:54.330424    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d474316d-631a-4b79-baaf-2e19fb16a211-istio-data" pod "d474316d-631a-4b79-baaf-2e19fb16a211" (UID: "d474316d-631a-4b79-baaf-2e19fb16a211"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:58:54.330628 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:54.330593    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d474316d-631a-4b79-baaf-2e19fb16a211-credential-socket" pod "d474316d-631a-4b79-baaf-2e19fb16a211" (UID: "d474316d-631a-4b79-baaf-2e19fb16a211"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:58:54.330750 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:54.330647    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d474316d-631a-4b79-baaf-2e19fb16a211-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d474316d-631a-4b79-baaf-2e19fb16a211-istiod-ca-cert\") pod \"d474316d-631a-4b79-baaf-2e19fb16a211\" (UID: \"d474316d-631a-4b79-baaf-2e19fb16a211\") "
Oct 30 13:58:54.330817 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:54.330745    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d474316d-631a-4b79-baaf-2e19fb16a211-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d474316d-631a-4b79-baaf-2e19fb16a211-workload-certs\") pod \"d474316d-631a-4b79-baaf-2e19fb16a211\" (UID: \"d474316d-631a-4b79-baaf-2e19fb16a211\") "
Oct 30 13:58:54.330817 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:54.330779    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d474316d-631a-4b79-baaf-2e19fb16a211-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d474316d-631a-4b79-baaf-2e19fb16a211-istio-podinfo\") pod \"d474316d-631a-4b79-baaf-2e19fb16a211\" (UID: \"d474316d-631a-4b79-baaf-2e19fb16a211\") "
Oct 30 13:58:54.330924 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:54.330880    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d474316d-631a-4b79-baaf-2e19fb16a211-istiod-ca-cert" pod "d474316d-631a-4b79-baaf-2e19fb16a211" (UID: "d474316d-631a-4b79-baaf-2e19fb16a211"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:58:54.331047 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:54.331011    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d474316d-631a-4b79-baaf-2e19fb16a211-workload-certs" pod "d474316d-631a-4b79-baaf-2e19fb16a211" (UID: "d474316d-631a-4b79-baaf-2e19fb16a211"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:58:54.331709 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:54.331682    5093 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d474316d-631a-4b79-baaf-2e19fb16a211-workload-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:58:54.331841 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:54.331713    5093 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d474316d-631a-4b79-baaf-2e19fb16a211-istio-data\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:58:54.331841 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:54.331726    5093 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d474316d-631a-4b79-baaf-2e19fb16a211-credential-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:58:54.331841 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:54.331741    5093 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d474316d-631a-4b79-baaf-2e19fb16a211-istiod-ca-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:58:54.331841 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:54.331754    5093 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d474316d-631a-4b79-baaf-2e19fb16a211-workload-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:58:54.332777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:54.332609    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d474316d-631a-4b79-baaf-2e19fb16a211-istio-token" pod "d474316d-631a-4b79-baaf-2e19fb16a211" (UID: "d474316d-631a-4b79-baaf-2e19fb16a211"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:58:54.332777 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:54.332736    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d474316d-631a-4b79-baaf-2e19fb16a211-kube-api-access-skms5" pod "d474316d-631a-4b79-baaf-2e19fb16a211" (UID: "d474316d-631a-4b79-baaf-2e19fb16a211"). InnerVolumeSpecName "kube-api-access-skms5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:58:54.333299 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:54.333275    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d474316d-631a-4b79-baaf-2e19fb16a211-istio-podinfo" pod "d474316d-631a-4b79-baaf-2e19fb16a211" (UID: "d474316d-631a-4b79-baaf-2e19fb16a211"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:58:54.333577 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:54.333558    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d474316d-631a-4b79-baaf-2e19fb16a211-istio-envoy" pod "d474316d-631a-4b79-baaf-2e19fb16a211" (UID: "d474316d-631a-4b79-baaf-2e19fb16a211"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:58:54.433647 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:54.433611    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d474316d-631a-4b79-baaf-2e19fb16a211-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:58:54.433647 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:54.433642    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skms5\" (UniqueName: \"kubernetes.io/projected/d474316d-631a-4b79-baaf-2e19fb16a211-kube-api-access-skms5\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:58:54.433647 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:54.433654    5093 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d474316d-631a-4b79-baaf-2e19fb16a211-istio-envoy\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:58:54.433953 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:54.433665    5093 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d474316d-631a-4b79-baaf-2e19fb16a211-istio-podinfo\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:58:54.803639 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:54.803546    5093 generic.go:356] "Generic (PLEG): container finished" podID="d474316d-631a-4b79-baaf-2e19fb16a211" containerID="b979fd59d3b2a04520ca52b9c57af7f10db30c394df8ba2a4d993b8fdab4e8d3" exitCode=0
Oct 30 13:58:54.803639 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:54.803608    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx"
Oct 30 13:58:54.803639 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:54.803609    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx" event={"ID":"d474316d-631a-4b79-baaf-2e19fb16a211","Type":"ContainerDied","Data":"b979fd59d3b2a04520ca52b9c57af7f10db30c394df8ba2a4d993b8fdab4e8d3"}
Oct 30 13:58:54.803906 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:54.803663    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx" event={"ID":"d474316d-631a-4b79-baaf-2e19fb16a211","Type":"ContainerDied","Data":"744acaf468401f798855d3d315d0ec42685bd784c8274a1fbf4dbdd316112a8e"}
Oct 30 13:58:54.803906 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:54.803695    5093 scope.go:122] "RemoveContainer" containerID="b979fd59d3b2a04520ca52b9c57af7f10db30c394df8ba2a4d993b8fdab4e8d3"
Oct 30 13:58:54.816838 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:54.816817    5093 scope.go:122] "RemoveContainer" containerID="ef9cc275b940404c3b166458727f4ead429a04c9b6dc15604b44792bf5a197ad"
Oct 30 13:58:54.829286 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:54.829259    5093 scope.go:122] "RemoveContainer" containerID="b979fd59d3b2a04520ca52b9c57af7f10db30c394df8ba2a4d993b8fdab4e8d3"
Oct 30 13:58:54.829632 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:58:54.829600    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b979fd59d3b2a04520ca52b9c57af7f10db30c394df8ba2a4d993b8fdab4e8d3\": container with ID starting with b979fd59d3b2a04520ca52b9c57af7f10db30c394df8ba2a4d993b8fdab4e8d3 not found: ID does not exist" containerID="b979fd59d3b2a04520ca52b9c57af7f10db30c394df8ba2a4d993b8fdab4e8d3"
Oct 30 13:58:54.829749 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:54.829637    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b979fd59d3b2a04520ca52b9c57af7f10db30c394df8ba2a4d993b8fdab4e8d3"} err="failed to get container status \"b979fd59d3b2a04520ca52b9c57af7f10db30c394df8ba2a4d993b8fdab4e8d3\": rpc error: code = NotFound desc = could not find container \"b979fd59d3b2a04520ca52b9c57af7f10db30c394df8ba2a4d993b8fdab4e8d3\": container with ID starting with b979fd59d3b2a04520ca52b9c57af7f10db30c394df8ba2a4d993b8fdab4e8d3 not found: ID does not exist"
Oct 30 13:58:54.829749 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:54.829656    5093 scope.go:122] "RemoveContainer" containerID="ef9cc275b940404c3b166458727f4ead429a04c9b6dc15604b44792bf5a197ad"
Oct 30 13:58:54.829939 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:58:54.829915    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef9cc275b940404c3b166458727f4ead429a04c9b6dc15604b44792bf5a197ad\": container with ID starting with ef9cc275b940404c3b166458727f4ead429a04c9b6dc15604b44792bf5a197ad not found: ID does not exist" containerID="ef9cc275b940404c3b166458727f4ead429a04c9b6dc15604b44792bf5a197ad"
Oct 30 13:58:54.829983 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:54.829953    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef9cc275b940404c3b166458727f4ead429a04c9b6dc15604b44792bf5a197ad"} err="failed to get container status \"ef9cc275b940404c3b166458727f4ead429a04c9b6dc15604b44792bf5a197ad\": rpc error: code = NotFound desc = could not find container \"ef9cc275b940404c3b166458727f4ead429a04c9b6dc15604b44792bf5a197ad\": container with ID starting with ef9cc275b940404c3b166458727f4ead429a04c9b6dc15604b44792bf5a197ad not found: ID does not exist"
Oct 30 13:58:54.893779 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:54.885032    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx"]
Oct 30 13:58:54.897918 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:54.897875    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-5494c9-openshift-default-5ccfbdc77f-fwsvx"]
Oct 30 13:58:55.002062 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:58:55.002017    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d474316d-631a-4b79-baaf-2e19fb16a211" path="/var/lib/kubelet/pods/d474316d-631a-4b79-baaf-2e19fb16a211/volumes"
Oct 30 13:59:13.756973 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:13.756684    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v"]
Oct 30 13:59:13.757910 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:13.757842    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v" podUID="4248f1a1-17bb-439c-80e5-8952de6299eb" containerName="istio-proxy" containerID="cri-o://adcee79ecc2104e6d28b6da391753c52c13369052f24f674b818701afbb12864" gracePeriod=30
Oct 30 13:59:19.035401 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:19.035308    5093 generic.go:356] "Generic (PLEG): container finished" podID="4248f1a1-17bb-439c-80e5-8952de6299eb" containerID="adcee79ecc2104e6d28b6da391753c52c13369052f24f674b818701afbb12864" exitCode=0
Oct 30 13:59:19.035866 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:19.035464    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v" event={"ID":"4248f1a1-17bb-439c-80e5-8952de6299eb","Type":"ContainerDied","Data":"adcee79ecc2104e6d28b6da391753c52c13369052f24f674b818701afbb12864"}
Oct 30 13:59:19.035866 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:19.035518    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v" event={"ID":"4248f1a1-17bb-439c-80e5-8952de6299eb","Type":"ContainerDied","Data":"345c47f7f87892c5b64275340275810c9916c7e37fff9f5031e62cc36c26212e"}
Oct 30 13:59:19.035866 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:19.035546    5093 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="345c47f7f87892c5b64275340275810c9916c7e37fff9f5031e62cc36c26212e"
Oct 30 13:59:19.035866 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:19.035564    5093 scope.go:122] "RemoveContainer" containerID="fb026cb86cadcf995e83773f4ace93a70ae80e522d3f5db954cae066427f38cf"
Oct 30 13:59:19.071225 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:19.071195    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v"
Oct 30 13:59:19.178038 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:19.178004    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4248f1a1-17bb-439c-80e5-8952de6299eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4248f1a1-17bb-439c-80e5-8952de6299eb-istio-podinfo\") pod \"4248f1a1-17bb-439c-80e5-8952de6299eb\" (UID: \"4248f1a1-17bb-439c-80e5-8952de6299eb\") "
Oct 30 13:59:19.178271 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:19.178047    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4248f1a1-17bb-439c-80e5-8952de6299eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4248f1a1-17bb-439c-80e5-8952de6299eb-workload-socket\") pod \"4248f1a1-17bb-439c-80e5-8952de6299eb\" (UID: \"4248f1a1-17bb-439c-80e5-8952de6299eb\") "
Oct 30 13:59:19.178271 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:19.178145    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4248f1a1-17bb-439c-80e5-8952de6299eb-kube-api-access-g8xxl\" (UniqueName: \"kubernetes.io/projected/4248f1a1-17bb-439c-80e5-8952de6299eb-kube-api-access-g8xxl\") pod \"4248f1a1-17bb-439c-80e5-8952de6299eb\" (UID: \"4248f1a1-17bb-439c-80e5-8952de6299eb\") "
Oct 30 13:59:19.178271 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:19.178204    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4248f1a1-17bb-439c-80e5-8952de6299eb-istio-token\" (UniqueName: \"kubernetes.io/projected/4248f1a1-17bb-439c-80e5-8952de6299eb-istio-token\") pod \"4248f1a1-17bb-439c-80e5-8952de6299eb\" (UID: \"4248f1a1-17bb-439c-80e5-8952de6299eb\") "
Oct 30 13:59:19.178411 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:19.178395    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4248f1a1-17bb-439c-80e5-8952de6299eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4248f1a1-17bb-439c-80e5-8952de6299eb-workload-certs\") pod \"4248f1a1-17bb-439c-80e5-8952de6299eb\" (UID: \"4248f1a1-17bb-439c-80e5-8952de6299eb\") "
Oct 30 13:59:19.178461 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:19.178428    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4248f1a1-17bb-439c-80e5-8952de6299eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4248f1a1-17bb-439c-80e5-8952de6299eb-istio-envoy\") pod \"4248f1a1-17bb-439c-80e5-8952de6299eb\" (UID: \"4248f1a1-17bb-439c-80e5-8952de6299eb\") "
Oct 30 13:59:19.178518 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:19.178454    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4248f1a1-17bb-439c-80e5-8952de6299eb-workload-socket" pod "4248f1a1-17bb-439c-80e5-8952de6299eb" (UID: "4248f1a1-17bb-439c-80e5-8952de6299eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:59:19.178518 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:19.178504    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4248f1a1-17bb-439c-80e5-8952de6299eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4248f1a1-17bb-439c-80e5-8952de6299eb-istio-data\") pod \"4248f1a1-17bb-439c-80e5-8952de6299eb\" (UID: \"4248f1a1-17bb-439c-80e5-8952de6299eb\") "
Oct 30 13:59:19.178641 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:19.178572    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4248f1a1-17bb-439c-80e5-8952de6299eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4248f1a1-17bb-439c-80e5-8952de6299eb-istiod-ca-cert\") pod \"4248f1a1-17bb-439c-80e5-8952de6299eb\" (UID: \"4248f1a1-17bb-439c-80e5-8952de6299eb\") "
Oct 30 13:59:19.178641 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:19.178616    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4248f1a1-17bb-439c-80e5-8952de6299eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4248f1a1-17bb-439c-80e5-8952de6299eb-credential-socket\") pod \"4248f1a1-17bb-439c-80e5-8952de6299eb\" (UID: \"4248f1a1-17bb-439c-80e5-8952de6299eb\") "
Oct 30 13:59:19.179010 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:19.178936    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4248f1a1-17bb-439c-80e5-8952de6299eb-workload-certs" pod "4248f1a1-17bb-439c-80e5-8952de6299eb" (UID: "4248f1a1-17bb-439c-80e5-8952de6299eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:59:19.179217 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:19.179175    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4248f1a1-17bb-439c-80e5-8952de6299eb-istiod-ca-cert" pod "4248f1a1-17bb-439c-80e5-8952de6299eb" (UID: "4248f1a1-17bb-439c-80e5-8952de6299eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:59:19.179526 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:19.179493    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4248f1a1-17bb-439c-80e5-8952de6299eb-credential-socket" pod "4248f1a1-17bb-439c-80e5-8952de6299eb" (UID: "4248f1a1-17bb-439c-80e5-8952de6299eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:59:19.179646 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:19.179594    5093 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4248f1a1-17bb-439c-80e5-8952de6299eb-workload-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:59:19.179646 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:19.179619    5093 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4248f1a1-17bb-439c-80e5-8952de6299eb-istiod-ca-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:59:19.179646 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:19.179633    5093 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4248f1a1-17bb-439c-80e5-8952de6299eb-workload-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:59:19.179814 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:19.179693    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4248f1a1-17bb-439c-80e5-8952de6299eb-istio-data" pod "4248f1a1-17bb-439c-80e5-8952de6299eb" (UID: "4248f1a1-17bb-439c-80e5-8952de6299eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:59:19.180928 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:19.180891    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4248f1a1-17bb-439c-80e5-8952de6299eb-kube-api-access-g8xxl" pod "4248f1a1-17bb-439c-80e5-8952de6299eb" (UID: "4248f1a1-17bb-439c-80e5-8952de6299eb"). InnerVolumeSpecName "kube-api-access-g8xxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:59:19.181230 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:19.181203    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4248f1a1-17bb-439c-80e5-8952de6299eb-istio-podinfo" pod "4248f1a1-17bb-439c-80e5-8952de6299eb" (UID: "4248f1a1-17bb-439c-80e5-8952de6299eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:59:19.181823 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:19.181804    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4248f1a1-17bb-439c-80e5-8952de6299eb-istio-token" pod "4248f1a1-17bb-439c-80e5-8952de6299eb" (UID: "4248f1a1-17bb-439c-80e5-8952de6299eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:59:19.181920 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:19.181905    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4248f1a1-17bb-439c-80e5-8952de6299eb-istio-envoy" pod "4248f1a1-17bb-439c-80e5-8952de6299eb" (UID: "4248f1a1-17bb-439c-80e5-8952de6299eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:59:19.281325 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:19.281282    5093 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4248f1a1-17bb-439c-80e5-8952de6299eb-istio-envoy\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:59:19.281325 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:19.281314    5093 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4248f1a1-17bb-439c-80e5-8952de6299eb-istio-data\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:59:19.281325 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:19.281327    5093 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4248f1a1-17bb-439c-80e5-8952de6299eb-credential-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:59:19.281575 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:19.281341    5093 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4248f1a1-17bb-439c-80e5-8952de6299eb-istio-podinfo\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:59:19.281575 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:19.281354    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8xxl\" (UniqueName: \"kubernetes.io/projected/4248f1a1-17bb-439c-80e5-8952de6299eb-kube-api-access-g8xxl\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:59:19.281575 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:19.281366    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4248f1a1-17bb-439c-80e5-8952de6299eb-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:59:20.045083 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:20.045050    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v"
Oct 30 13:59:20.148007 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:20.147972    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v"]
Oct 30 13:59:20.152294 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:20.152266    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-51605e-openshift-default-fb8999c9f-ckc4v"]
Oct 30 13:59:21.005979 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:21.005940    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4248f1a1-17bb-439c-80e5-8952de6299eb" path="/var/lib/kubelet/pods/4248f1a1-17bb-439c-80e5-8952de6299eb/volumes"
Oct 30 13:59:21.202151 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:59:21.201554    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 13:59:46.703427 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:46.703381    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-08c365/test-model-08c365-kserve-576d588df-fsfnh"]
Oct 30 13:59:48.801889 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:48.801850    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-996b8b-56c78549fc-t7vl6"]
Oct 30 13:59:48.802460 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:48.802177    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-996b8b-56c78549fc-t7vl6" podUID="dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412" containerName="maas-api" containerID="cri-o://682ee22f0f19cdad84b8eb4c2260f157f2096988799597c6e0242f45240d0456" gracePeriod=30
Oct 30 13:59:49.161245 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:49.161210    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-996b8b-56c78549fc-t7vl6"
Oct 30 13:59:49.210960 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:49.210919    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412-maas-api-tls\") pod \"dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412\" (UID: \"dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412\") "
Oct 30 13:59:49.211236 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:49.211077    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412-openshift-service-ca\") pod \"dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412\" (UID: \"dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412\") "
Oct 30 13:59:49.211236 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:49.211119    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412-kube-api-access-5j9nz\" (UniqueName: \"kubernetes.io/projected/dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412-kube-api-access-5j9nz\") pod \"dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412\" (UID: \"dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412\") "
Oct 30 13:59:49.211236 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:49.211232    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412-metrics-certs\") pod \"dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412\" (UID: \"dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412\") "
Oct 30 13:59:49.212298 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:49.211681    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412-openshift-service-ca" pod "dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412" (UID: "dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:59:49.212298 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:49.211880    5093 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412-openshift-service-ca\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:59:49.217303 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:49.216094    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412-metrics-certs" pod "dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412" (UID: "dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:59:49.217303 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:49.216247    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412-maas-api-tls" pod "dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412" (UID: "dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:59:49.217429 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:49.217305    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412-kube-api-access-5j9nz" pod "dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412" (UID: "dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412"). InnerVolumeSpecName "kube-api-access-5j9nz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:59:49.320160 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:49.319984    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j9nz\" (UniqueName: \"kubernetes.io/projected/dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412-kube-api-access-5j9nz\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:59:49.320160 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:49.320024    5093 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412-metrics-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:59:49.320160 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:49.320039    5093 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412-maas-api-tls\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 13:59:49.342161 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:49.341718    5093 generic.go:356] "Generic (PLEG): container finished" podID="dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412" containerID="682ee22f0f19cdad84b8eb4c2260f157f2096988799597c6e0242f45240d0456" exitCode=0
Oct 30 13:59:49.342161 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:49.342047    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-996b8b-56c78549fc-t7vl6" event={"ID":"dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412","Type":"ContainerDied","Data":"682ee22f0f19cdad84b8eb4c2260f157f2096988799597c6e0242f45240d0456"}
Oct 30 13:59:49.342161 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:49.342099    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-996b8b-56c78549fc-t7vl6" event={"ID":"dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412","Type":"ContainerDied","Data":"4530d9cab6021e85ecd7edfdd9ccd3df7cbd5e1bfe7023c9545bda1184ef15e3"}
Oct 30 13:59:49.342161 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:49.342146    5093 scope.go:122] "RemoveContainer" containerID="682ee22f0f19cdad84b8eb4c2260f157f2096988799597c6e0242f45240d0456"
Oct 30 13:59:49.342563 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:49.342305    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-996b8b-56c78549fc-t7vl6"
Oct 30 13:59:49.373344 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:49.373320    5093 scope.go:122] "RemoveContainer" containerID="682ee22f0f19cdad84b8eb4c2260f157f2096988799597c6e0242f45240d0456"
Oct 30 13:59:49.373735 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:59:49.373708    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"682ee22f0f19cdad84b8eb4c2260f157f2096988799597c6e0242f45240d0456\": container with ID starting with 682ee22f0f19cdad84b8eb4c2260f157f2096988799597c6e0242f45240d0456 not found: ID does not exist" containerID="682ee22f0f19cdad84b8eb4c2260f157f2096988799597c6e0242f45240d0456"
Oct 30 13:59:49.373857 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:49.373747    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"682ee22f0f19cdad84b8eb4c2260f157f2096988799597c6e0242f45240d0456"} err="failed to get container status \"682ee22f0f19cdad84b8eb4c2260f157f2096988799597c6e0242f45240d0456\": rpc error: code = NotFound desc = could not find container \"682ee22f0f19cdad84b8eb4c2260f157f2096988799597c6e0242f45240d0456\": container with ID starting with 682ee22f0f19cdad84b8eb4c2260f157f2096988799597c6e0242f45240d0456 not found: ID does not exist"
Oct 30 13:59:49.405425 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:49.405374    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-996b8b-56c78549fc-t7vl6"]
Oct 30 13:59:49.411972 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:49.411934    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-996b8b-56c78549fc-t7vl6"]
Oct 30 13:59:51.001425 ip-10-0-134-31 kubenswrapper[5093]: I1030 13:59:51.001381    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412" path="/var/lib/kubelet/pods/dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412/volumes"
Oct 30 13:59:51.168990 ip-10-0-134-31 kubenswrapper[5093]: E1030 13:59:51.168946    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 14:00:00.280705 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.280663    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889480-qlw8p"]
Oct 30 14:00:00.281575 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.281554    5093 cpu_manager.go:414] "containerMap: removing container" podUID="d474316d-631a-4b79-baaf-2e19fb16a211" containerName="istio-proxy"
Oct 30 14:00:00.281575 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.281575    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d474316d-631a-4b79-baaf-2e19fb16a211" containerName="istio-proxy"
Oct 30 14:00:00.281696 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.281586    5093 cpu_manager.go:414] "containerMap: removing container" podUID="d474316d-631a-4b79-baaf-2e19fb16a211" containerName="istio-proxy"
Oct 30 14:00:00.281696 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.281591    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d474316d-631a-4b79-baaf-2e19fb16a211" containerName="istio-proxy"
Oct 30 14:00:00.281696 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.281606    5093 cpu_manager.go:414] "containerMap: removing container" podUID="1a9a0d53-3fb0-414a-a543-f5dd4ed39eef" containerName="istio-proxy"
Oct 30 14:00:00.281696 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.281612    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a9a0d53-3fb0-414a-a543-f5dd4ed39eef" containerName="istio-proxy"
Oct 30 14:00:00.281696 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.281628    5093 cpu_manager.go:414] "containerMap: removing container" podUID="4248f1a1-17bb-439c-80e5-8952de6299eb" containerName="istio-proxy"
Oct 30 14:00:00.281696 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.281633    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4248f1a1-17bb-439c-80e5-8952de6299eb" containerName="istio-proxy"
Oct 30 14:00:00.281696 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.281643    5093 cpu_manager.go:414] "containerMap: removing container" podUID="14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5" containerName="main"
Oct 30 14:00:00.281696 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.281647    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5" containerName="main"
Oct 30 14:00:00.281696 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.281660    5093 cpu_manager.go:414] "containerMap: removing container" podUID="1a9a0d53-3fb0-414a-a543-f5dd4ed39eef" containerName="istio-proxy"
Oct 30 14:00:00.281696 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.281665    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a9a0d53-3fb0-414a-a543-f5dd4ed39eef" containerName="istio-proxy"
Oct 30 14:00:00.281696 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.281680    5093 cpu_manager.go:414] "containerMap: removing container" podUID="4248f1a1-17bb-439c-80e5-8952de6299eb" containerName="istio-proxy"
Oct 30 14:00:00.281696 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.281685    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4248f1a1-17bb-439c-80e5-8952de6299eb" containerName="istio-proxy"
Oct 30 14:00:00.281696 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.281699    5093 cpu_manager.go:414] "containerMap: removing container" podUID="dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412" containerName="maas-api"
Oct 30 14:00:00.281696 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.281704    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412" containerName="maas-api"
Oct 30 14:00:00.282564 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.281711    5093 cpu_manager.go:414] "containerMap: removing container" podUID="8aaca537-18ae-4b3a-9da2-3c834d740100" containerName="maas-api"
Oct 30 14:00:00.282564 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.281716    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aaca537-18ae-4b3a-9da2-3c834d740100" containerName="maas-api"
Oct 30 14:00:00.282564 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.281828    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="14eb7d03-231f-46f2-aa9c-9f00a1e9a3b5" containerName="main"
Oct 30 14:00:00.282564 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.281839    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a9a0d53-3fb0-414a-a543-f5dd4ed39eef" containerName="istio-proxy"
Oct 30 14:00:00.282564 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.281846    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="4248f1a1-17bb-439c-80e5-8952de6299eb" containerName="istio-proxy"
Oct 30 14:00:00.282564 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.281853    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="dead8a2f-bc4e-4ae2-aac0-bcb0dd28a412" containerName="maas-api"
Oct 30 14:00:00.282564 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.281859    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aaca537-18ae-4b3a-9da2-3c834d740100" containerName="maas-api"
Oct 30 14:00:00.282564 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.281868    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="d474316d-631a-4b79-baaf-2e19fb16a211" containerName="istio-proxy"
Oct 30 14:00:00.282564 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.281876    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="d474316d-631a-4b79-baaf-2e19fb16a211" containerName="istio-proxy"
Oct 30 14:00:00.282564 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.281882    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a9a0d53-3fb0-414a-a543-f5dd4ed39eef" containerName="istio-proxy"
Oct 30 14:00:00.287551 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.287519    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889480-qlw8p"
Oct 30 14:00:00.306848 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.306799    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889480-lxfxn"]
Oct 30 14:00:00.308745 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.308711    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="4248f1a1-17bb-439c-80e5-8952de6299eb" containerName="istio-proxy"
Oct 30 14:00:00.316872 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.316457    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889480-lxfxn"
Oct 30 14:00:00.321514 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.321461    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 14:00:00.325014 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.324976    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889480-qlw8p"]
Oct 30 14:00:00.349548 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.347414    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889480-lxfxn"]
Oct 30 14:00:00.359318 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.359276    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889480-r8ddx"]
Oct 30 14:00:00.364779 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.364747    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889480-r8ddx"
Oct 30 14:00:00.369072 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.366614    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889480-hl64w"]
Oct 30 14:00:00.373679 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.373253    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889480-hl64w"
Oct 30 14:00:00.397791 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.397620    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889480-r8ddx"]
Oct 30 14:00:00.404088 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.404051    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba607a11-313e-4fd8-b812-771fd8dbbf92-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889480-qlw8p\" (UID: \"ba607a11-313e-4fd8-b812-771fd8dbbf92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889480-qlw8p"
Oct 30 14:00:00.405688 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.405368    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46crf\" (UniqueName: \"kubernetes.io/projected/ba607a11-313e-4fd8-b812-771fd8dbbf92-kube-api-access-46crf\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889480-qlw8p\" (UID: \"ba607a11-313e-4fd8-b812-771fd8dbbf92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889480-qlw8p"
Oct 30 14:00:00.409581 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.409556    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2rjh\" (UniqueName: \"kubernetes.io/projected/073758c5-f33b-459c-a0a7-040c013d39db-kube-api-access-l2rjh\") pod \"maas-api-key-cleanup-e2e-worker-w1-52af49-29889480-lxfxn\" (UID: \"073758c5-f33b-459c-a0a7-040c013d39db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889480-lxfxn"
Oct 30 14:00:00.409887 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.409865    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/073758c5-f33b-459c-a0a7-040c013d39db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-52af49-29889480-lxfxn\" (UID: \"073758c5-f33b-459c-a0a7-040c013d39db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889480-lxfxn"
Oct 30 14:00:00.410377 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.406732    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889480-vwdrd"]
Oct 30 14:00:00.420379 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.420342    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889480-hl64w"]
Oct 30 14:00:00.420552 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.420471    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889480-vwdrd"
Oct 30 14:00:00.475298 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.475246    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889480-vwdrd"]
Oct 30 14:00:00.511486 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.511443    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncwtz\" (UniqueName: \"kubernetes.io/projected/992c57a3-547b-4b38-bdcf-139eb477dcbc-kube-api-access-ncwtz\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889480-vwdrd\" (UID: \"992c57a3-547b-4b38-bdcf-139eb477dcbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889480-vwdrd"
Oct 30 14:00:00.511797 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.511768    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/992c57a3-547b-4b38-bdcf-139eb477dcbc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889480-vwdrd\" (UID: \"992c57a3-547b-4b38-bdcf-139eb477dcbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889480-vwdrd"
Oct 30 14:00:00.512001 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.511982    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssbbt\" (UniqueName: \"kubernetes.io/projected/41cca4c3-513c-4628-997d-545b0326a9ac-kube-api-access-ssbbt\") pod \"maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889480-hl64w\" (UID: \"41cca4c3-513c-4628-997d-545b0326a9ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889480-hl64w"
Oct 30 14:00:00.512255 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.512042    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/073758c5-f33b-459c-a0a7-040c013d39db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-52af49-29889480-lxfxn\" (UID: \"073758c5-f33b-459c-a0a7-040c013d39db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889480-lxfxn"
Oct 30 14:00:00.512255 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.512174    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbeb73e2-4d19-404a-9789-b8828f2b2ed4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889480-r8ddx\" (UID: \"dbeb73e2-4d19-404a-9789-b8828f2b2ed4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889480-r8ddx"
Oct 30 14:00:00.512411 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.512245    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2thg\" (UniqueName: \"kubernetes.io/projected/dbeb73e2-4d19-404a-9789-b8828f2b2ed4-kube-api-access-q2thg\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889480-r8ddx\" (UID: \"dbeb73e2-4d19-404a-9789-b8828f2b2ed4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889480-r8ddx"
Oct 30 14:00:00.512411 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.512356    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41cca4c3-513c-4628-997d-545b0326a9ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889480-hl64w\" (UID: \"41cca4c3-513c-4628-997d-545b0326a9ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889480-hl64w"
Oct 30 14:00:00.512524 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.512443    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba607a11-313e-4fd8-b812-771fd8dbbf92-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889480-qlw8p\" (UID: \"ba607a11-313e-4fd8-b812-771fd8dbbf92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889480-qlw8p"
Oct 30 14:00:00.512524 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.512492    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46crf\" (UniqueName: \"kubernetes.io/projected/ba607a11-313e-4fd8-b812-771fd8dbbf92-kube-api-access-46crf\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889480-qlw8p\" (UID: \"ba607a11-313e-4fd8-b812-771fd8dbbf92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889480-qlw8p"
Oct 30 14:00:00.512642 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.512520    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2rjh\" (UniqueName: \"kubernetes.io/projected/073758c5-f33b-459c-a0a7-040c013d39db-kube-api-access-l2rjh\") pod \"maas-api-key-cleanup-e2e-worker-w1-52af49-29889480-lxfxn\" (UID: \"073758c5-f33b-459c-a0a7-040c013d39db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889480-lxfxn"
Oct 30 14:00:00.512933 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.512911    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/073758c5-f33b-459c-a0a7-040c013d39db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-52af49-29889480-lxfxn\" (UID: \"073758c5-f33b-459c-a0a7-040c013d39db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889480-lxfxn"
Oct 30 14:00:00.513601 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.513578    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba607a11-313e-4fd8-b812-771fd8dbbf92-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889480-qlw8p\" (UID: \"ba607a11-313e-4fd8-b812-771fd8dbbf92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889480-qlw8p"
Oct 30 14:00:00.538605 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.538527    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2rjh\" (UniqueName: \"kubernetes.io/projected/073758c5-f33b-459c-a0a7-040c013d39db-kube-api-access-l2rjh\") pod \"maas-api-key-cleanup-e2e-worker-w1-52af49-29889480-lxfxn\" (UID: \"073758c5-f33b-459c-a0a7-040c013d39db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889480-lxfxn"
Oct 30 14:00:00.538775 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.538748    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46crf\" (UniqueName: \"kubernetes.io/projected/ba607a11-313e-4fd8-b812-771fd8dbbf92-kube-api-access-46crf\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889480-qlw8p\" (UID: \"ba607a11-313e-4fd8-b812-771fd8dbbf92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889480-qlw8p"
Oct 30 14:00:00.609539 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.609502    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889480-qlw8p"
Oct 30 14:00:00.613430 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.613347    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/992c57a3-547b-4b38-bdcf-139eb477dcbc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889480-vwdrd\" (UID: \"992c57a3-547b-4b38-bdcf-139eb477dcbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889480-vwdrd"
Oct 30 14:00:00.613430 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.613397    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssbbt\" (UniqueName: \"kubernetes.io/projected/41cca4c3-513c-4628-997d-545b0326a9ac-kube-api-access-ssbbt\") pod \"maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889480-hl64w\" (UID: \"41cca4c3-513c-4628-997d-545b0326a9ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889480-hl64w"
Oct 30 14:00:00.613578 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.613481    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbeb73e2-4d19-404a-9789-b8828f2b2ed4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889480-r8ddx\" (UID: \"dbeb73e2-4d19-404a-9789-b8828f2b2ed4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889480-r8ddx"
Oct 30 14:00:00.613578 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.613511    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2thg\" (UniqueName: \"kubernetes.io/projected/dbeb73e2-4d19-404a-9789-b8828f2b2ed4-kube-api-access-q2thg\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889480-r8ddx\" (UID: \"dbeb73e2-4d19-404a-9789-b8828f2b2ed4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889480-r8ddx"
Oct 30 14:00:00.613578 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.613571    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41cca4c3-513c-4628-997d-545b0326a9ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889480-hl64w\" (UID: \"41cca4c3-513c-4628-997d-545b0326a9ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889480-hl64w"
Oct 30 14:00:00.613757 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.613680    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncwtz\" (UniqueName: \"kubernetes.io/projected/992c57a3-547b-4b38-bdcf-139eb477dcbc-kube-api-access-ncwtz\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889480-vwdrd\" (UID: \"992c57a3-547b-4b38-bdcf-139eb477dcbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889480-vwdrd"
Oct 30 14:00:00.614202 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.614168    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/992c57a3-547b-4b38-bdcf-139eb477dcbc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889480-vwdrd\" (UID: \"992c57a3-547b-4b38-bdcf-139eb477dcbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889480-vwdrd"
Oct 30 14:00:00.614313 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.614176    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbeb73e2-4d19-404a-9789-b8828f2b2ed4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889480-r8ddx\" (UID: \"dbeb73e2-4d19-404a-9789-b8828f2b2ed4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889480-r8ddx"
Oct 30 14:00:00.614407 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.614390    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41cca4c3-513c-4628-997d-545b0326a9ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889480-hl64w\" (UID: \"41cca4c3-513c-4628-997d-545b0326a9ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889480-hl64w"
Oct 30 14:00:00.635048 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.635000    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncwtz\" (UniqueName: \"kubernetes.io/projected/992c57a3-547b-4b38-bdcf-139eb477dcbc-kube-api-access-ncwtz\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889480-vwdrd\" (UID: \"992c57a3-547b-4b38-bdcf-139eb477dcbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889480-vwdrd"
Oct 30 14:00:00.635048 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.635045    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssbbt\" (UniqueName: \"kubernetes.io/projected/41cca4c3-513c-4628-997d-545b0326a9ac-kube-api-access-ssbbt\") pod \"maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889480-hl64w\" (UID: \"41cca4c3-513c-4628-997d-545b0326a9ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889480-hl64w"
Oct 30 14:00:00.635361 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.635087    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2thg\" (UniqueName: \"kubernetes.io/projected/dbeb73e2-4d19-404a-9789-b8828f2b2ed4-kube-api-access-q2thg\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889480-r8ddx\" (UID: \"dbeb73e2-4d19-404a-9789-b8828f2b2ed4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889480-r8ddx"
Oct 30 14:00:00.666580 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.666535    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889480-lxfxn"
Oct 30 14:00:00.715461 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.715418    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889480-r8ddx"
Oct 30 14:00:00.716651 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.715723    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889480-hl64w"
Oct 30 14:00:00.768447 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.767975    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889480-vwdrd"
Oct 30 14:00:00.945959 ip-10-0-134-31 kubenswrapper[5093]: W1030 14:00:00.945924    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba607a11_313e_4fd8_b812_771fd8dbbf92.slice/crio-001515d0cb9c7e4fa84e3ffdfc00e0ff2b662791c9f464994a5a4ee16885598b WatchSource:0}: Error finding container 001515d0cb9c7e4fa84e3ffdfc00e0ff2b662791c9f464994a5a4ee16885598b: Status 404 returned error can't find the container with id 001515d0cb9c7e4fa84e3ffdfc00e0ff2b662791c9f464994a5a4ee16885598b
Oct 30 14:00:00.947442 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:00.947419    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889480-qlw8p"]
Oct 30 14:00:01.174203 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:01.174173    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889480-vwdrd"]
Oct 30 14:00:01.175656 ip-10-0-134-31 kubenswrapper[5093]: W1030 14:00:01.175621    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod992c57a3_547b_4b38_bdcf_139eb477dcbc.slice/crio-ba71363ed811257007d2ebc2745b02d2bcb8fb2ee7f2747f0d6ce79b81ffe8c1 WatchSource:0}: Error finding container ba71363ed811257007d2ebc2745b02d2bcb8fb2ee7f2747f0d6ce79b81ffe8c1: Status 404 returned error can't find the container with id ba71363ed811257007d2ebc2745b02d2bcb8fb2ee7f2747f0d6ce79b81ffe8c1
Oct 30 14:00:01.239499 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:01.238396    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889480-lxfxn"]
Oct 30 14:00:01.242322 ip-10-0-134-31 kubenswrapper[5093]: W1030 14:00:01.242290    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod073758c5_f33b_459c_a0a7_040c013d39db.slice/crio-c2bb8ac86757c7d353207fc0c2134d901c29bd9799a9e704660ab78974646a46 WatchSource:0}: Error finding container c2bb8ac86757c7d353207fc0c2134d901c29bd9799a9e704660ab78974646a46: Status 404 returned error can't find the container with id c2bb8ac86757c7d353207fc0c2134d901c29bd9799a9e704660ab78974646a46
Oct 30 14:00:01.279084 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:01.279055    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889480-r8ddx"]
Oct 30 14:00:01.283544 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:01.283518    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889480-hl64w"]
Oct 30 14:00:01.284146 ip-10-0-134-31 kubenswrapper[5093]: W1030 14:00:01.284104    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbeb73e2_4d19_404a_9789_b8828f2b2ed4.slice/crio-21e44d6e104421ef16253942745e92ee7e56f516e68ab403b979bb5515180dab WatchSource:0}: Error finding container 21e44d6e104421ef16253942745e92ee7e56f516e68ab403b979bb5515180dab: Status 404 returned error can't find the container with id 21e44d6e104421ef16253942745e92ee7e56f516e68ab403b979bb5515180dab
Oct 30 14:00:01.284384 ip-10-0-134-31 kubenswrapper[5093]: W1030 14:00:01.284363    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41cca4c3_513c_4628_997d_545b0326a9ac.slice/crio-94aab6c9c5e74bde4e21561b235901f8410c1c174a63b95faa549b93a941fc25 WatchSource:0}: Error finding container 94aab6c9c5e74bde4e21561b235901f8410c1c174a63b95faa549b93a941fc25: Status 404 returned error can't find the container with id 94aab6c9c5e74bde4e21561b235901f8410c1c174a63b95faa549b93a941fc25
Oct 30 14:00:01.494593 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:01.494556    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889480-r8ddx" event={"ID":"dbeb73e2-4d19-404a-9789-b8828f2b2ed4","Type":"ContainerStarted","Data":"21e44d6e104421ef16253942745e92ee7e56f516e68ab403b979bb5515180dab"}
Oct 30 14:00:01.496166 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:01.496102    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889480-vwdrd" event={"ID":"992c57a3-547b-4b38-bdcf-139eb477dcbc","Type":"ContainerStarted","Data":"ba71363ed811257007d2ebc2745b02d2bcb8fb2ee7f2747f0d6ce79b81ffe8c1"}
Oct 30 14:00:01.497551 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:01.497518    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889480-hl64w" event={"ID":"41cca4c3-513c-4628-997d-545b0326a9ac","Type":"ContainerStarted","Data":"94aab6c9c5e74bde4e21561b235901f8410c1c174a63b95faa549b93a941fc25"}
Oct 30 14:00:01.499014 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:01.498990    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889480-lxfxn" event={"ID":"073758c5-f33b-459c-a0a7-040c013d39db","Type":"ContainerStarted","Data":"c2bb8ac86757c7d353207fc0c2134d901c29bd9799a9e704660ab78974646a46"}
Oct 30 14:00:01.500701 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:01.500678    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889480-qlw8p" event={"ID":"ba607a11-313e-4fd8-b812-771fd8dbbf92","Type":"ContainerStarted","Data":"9b7fee5a1751a15a9914a659d8a78909edbaf8c534552c79e9a34aa41170044a"}
Oct 30 14:00:01.500806 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:01.500722    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889480-qlw8p" event={"ID":"ba607a11-313e-4fd8-b812-771fd8dbbf92","Type":"ContainerStarted","Data":"001515d0cb9c7e4fa84e3ffdfc00e0ff2b662791c9f464994a5a4ee16885598b"}
Oct 30 14:00:01.548635 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:01.548563    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889480-qlw8p" podStartSLOduration=1.548542224 podStartE2EDuration="1.548542224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:00:01.545405217 +0000 UTC m=+3420.692672459" watchObservedRunningTime="2026-10-30 14:00:01.548542224 +0000 UTC m=+3420.695809465"
Oct 30 14:00:01.729512 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:01.728878    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-08c365/test-model-08c365-kserve-576d588df-fsfnh" podUID="2f73d682-50c2-46d3-a907-dc4c6fb089af" containerName="main" containerID="cri-o://f73f4c35aacad6329ca64a4a2f639d95cb54b06b6b9848d8ae2e55d0fca3060c" gracePeriod=45
Oct 30 14:00:01.907485 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:01.907437    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-08c365/test-model-08c365-kserve-576d588df-fsfnh" podUID="2f73d682-50c2-46d3-a907-dc4c6fb089af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.186:8000/ready\": dial tcp 10.132.0.186:8000: connect: connection refused"
Oct 30 14:00:02.134316 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:02.134244    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-08c365/test-model-08c365-kserve-576d588df-fsfnh"
Oct 30 14:00:02.163605 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:02.163557    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f73d682-50c2-46d3-a907-dc4c6fb089af-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f73d682-50c2-46d3-a907-dc4c6fb089af-tmp-dir\") pod \"2f73d682-50c2-46d3-a907-dc4c6fb089af\" (UID: \"2f73d682-50c2-46d3-a907-dc4c6fb089af\") "
Oct 30 14:00:02.163798 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:02.163720    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f73d682-50c2-46d3-a907-dc4c6fb089af-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f73d682-50c2-46d3-a907-dc4c6fb089af-model-cache\") pod \"2f73d682-50c2-46d3-a907-dc4c6fb089af\" (UID: \"2f73d682-50c2-46d3-a907-dc4c6fb089af\") "
Oct 30 14:00:02.163861 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:02.163794    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f73d682-50c2-46d3-a907-dc4c6fb089af-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f73d682-50c2-46d3-a907-dc4c6fb089af-dshm\") pod \"2f73d682-50c2-46d3-a907-dc4c6fb089af\" (UID: \"2f73d682-50c2-46d3-a907-dc4c6fb089af\") "
Oct 30 14:00:02.163861 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:02.163824    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f73d682-50c2-46d3-a907-dc4c6fb089af-home\" (UniqueName: \"kubernetes.io/empty-dir/2f73d682-50c2-46d3-a907-dc4c6fb089af-home\") pod \"2f73d682-50c2-46d3-a907-dc4c6fb089af\" (UID: \"2f73d682-50c2-46d3-a907-dc4c6fb089af\") "
Oct 30 14:00:02.163861 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:02.163855    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f73d682-50c2-46d3-a907-dc4c6fb089af-tls-certs\" (UniqueName: \"kubernetes.io/secret/2f73d682-50c2-46d3-a907-dc4c6fb089af-tls-certs\") pod \"2f73d682-50c2-46d3-a907-dc4c6fb089af\" (UID: \"2f73d682-50c2-46d3-a907-dc4c6fb089af\") "
Oct 30 14:00:02.164040 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:02.163869    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f73d682-50c2-46d3-a907-dc4c6fb089af-tmp-dir" pod "2f73d682-50c2-46d3-a907-dc4c6fb089af" (UID: "2f73d682-50c2-46d3-a907-dc4c6fb089af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:00:02.164100 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:02.164049    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f73d682-50c2-46d3-a907-dc4c6fb089af-kube-api-access-45nn9\" (UniqueName: \"kubernetes.io/projected/2f73d682-50c2-46d3-a907-dc4c6fb089af-kube-api-access-45nn9\") pod \"2f73d682-50c2-46d3-a907-dc4c6fb089af\" (UID: \"2f73d682-50c2-46d3-a907-dc4c6fb089af\") "
Oct 30 14:00:02.164291 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:02.164049    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f73d682-50c2-46d3-a907-dc4c6fb089af-model-cache" pod "2f73d682-50c2-46d3-a907-dc4c6fb089af" (UID: "2f73d682-50c2-46d3-a907-dc4c6fb089af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:00:02.164291 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:02.164234    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f73d682-50c2-46d3-a907-dc4c6fb089af-home" pod "2f73d682-50c2-46d3-a907-dc4c6fb089af" (UID: "2f73d682-50c2-46d3-a907-dc4c6fb089af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:00:02.164721 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:02.164700    5093 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f73d682-50c2-46d3-a907-dc4c6fb089af-tmp-dir\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:00:02.164837 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:02.164820    5093 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f73d682-50c2-46d3-a907-dc4c6fb089af-model-cache\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:00:02.164899 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:02.164840    5093 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f73d682-50c2-46d3-a907-dc4c6fb089af-home\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:00:02.166637 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:02.166616    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f73d682-50c2-46d3-a907-dc4c6fb089af-dshm" pod "2f73d682-50c2-46d3-a907-dc4c6fb089af" (UID: "2f73d682-50c2-46d3-a907-dc4c6fb089af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:00:02.166798 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:02.166776    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f73d682-50c2-46d3-a907-dc4c6fb089af-kube-api-access-45nn9" pod "2f73d682-50c2-46d3-a907-dc4c6fb089af" (UID: "2f73d682-50c2-46d3-a907-dc4c6fb089af"). InnerVolumeSpecName "kube-api-access-45nn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:00:02.166870 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:02.166782    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f73d682-50c2-46d3-a907-dc4c6fb089af-tls-certs" pod "2f73d682-50c2-46d3-a907-dc4c6fb089af" (UID: "2f73d682-50c2-46d3-a907-dc4c6fb089af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:00:02.265908 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:02.265827    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45nn9\" (UniqueName: \"kubernetes.io/projected/2f73d682-50c2-46d3-a907-dc4c6fb089af-kube-api-access-45nn9\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:00:02.265908 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:02.265862    5093 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f73d682-50c2-46d3-a907-dc4c6fb089af-dshm\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:00:02.265908 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:02.265872    5093 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f73d682-50c2-46d3-a907-dc4c6fb089af-tls-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:00:02.517280 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:02.517179    5093 generic.go:356] "Generic (PLEG): container finished" podID="ba607a11-313e-4fd8-b812-771fd8dbbf92" containerID="9b7fee5a1751a15a9914a659d8a78909edbaf8c534552c79e9a34aa41170044a" exitCode=60
Oct 30 14:00:02.517998 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:02.517964    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889480-qlw8p" event={"ID":"ba607a11-313e-4fd8-b812-771fd8dbbf92","Type":"ContainerDied","Data":"9b7fee5a1751a15a9914a659d8a78909edbaf8c534552c79e9a34aa41170044a"}
Oct 30 14:00:02.519179 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:02.517702    5093 scope.go:122] "RemoveContainer" containerID="9b7fee5a1751a15a9914a659d8a78909edbaf8c534552c79e9a34aa41170044a"
Oct 30 14:00:02.521424 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:02.521394    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889480-r8ddx" event={"ID":"dbeb73e2-4d19-404a-9789-b8828f2b2ed4","Type":"ContainerStarted","Data":"bdf8cd413b619c65a718f3be73a4a370464b237b3126da308b49d383da99a88d"}
Oct 30 14:00:02.529001 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:02.527974    5093 generic.go:356] "Generic (PLEG): container finished" podID="992c57a3-547b-4b38-bdcf-139eb477dcbc" containerID="ca3891d5e7c302b08d75aea53765eb5b26f0b1239de6a8da3b7de271a780cb7b" exitCode=60
Oct 30 14:00:02.529001 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:02.528109    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889480-vwdrd" event={"ID":"992c57a3-547b-4b38-bdcf-139eb477dcbc","Type":"ContainerDied","Data":"ca3891d5e7c302b08d75aea53765eb5b26f0b1239de6a8da3b7de271a780cb7b"}
Oct 30 14:00:02.529001 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:02.528524    5093 scope.go:122] "RemoveContainer" containerID="ca3891d5e7c302b08d75aea53765eb5b26f0b1239de6a8da3b7de271a780cb7b"
Oct 30 14:00:02.539081 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:02.539051    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889480-hl64w" event={"ID":"41cca4c3-513c-4628-997d-545b0326a9ac","Type":"ContainerStarted","Data":"c8b173302621d139cc605b760f93ff77a40162cd72be1fe7abf645c4c201dcb2"}
Oct 30 14:00:02.542103 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:02.542077    5093 generic.go:356] "Generic (PLEG): container finished" podID="073758c5-f33b-459c-a0a7-040c013d39db" containerID="1813bd5a67e909c43499c08c43d9cd58bd2e84c8365eb68b09a547d9b3484e67" exitCode=60
Oct 30 14:00:02.542227 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:02.542191    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889480-lxfxn" event={"ID":"073758c5-f33b-459c-a0a7-040c013d39db","Type":"ContainerDied","Data":"1813bd5a67e909c43499c08c43d9cd58bd2e84c8365eb68b09a547d9b3484e67"}
Oct 30 14:00:02.542455 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:02.542436    5093 scope.go:122] "RemoveContainer" containerID="1813bd5a67e909c43499c08c43d9cd58bd2e84c8365eb68b09a547d9b3484e67"
Oct 30 14:00:02.543765 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:02.543744    5093 generic.go:356] "Generic (PLEG): container finished" podID="2f73d682-50c2-46d3-a907-dc4c6fb089af" containerID="f73f4c35aacad6329ca64a4a2f639d95cb54b06b6b9848d8ae2e55d0fca3060c" exitCode=0
Oct 30 14:00:02.543862 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:02.543796    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-08c365/test-model-08c365-kserve-576d588df-fsfnh" event={"ID":"2f73d682-50c2-46d3-a907-dc4c6fb089af","Type":"ContainerDied","Data":"f73f4c35aacad6329ca64a4a2f639d95cb54b06b6b9848d8ae2e55d0fca3060c"}
Oct 30 14:00:02.543862 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:02.543825    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-08c365/test-model-08c365-kserve-576d588df-fsfnh" event={"ID":"2f73d682-50c2-46d3-a907-dc4c6fb089af","Type":"ContainerDied","Data":"c212b7e9fc5ac5fe787ed3c7318fae83dc1740f994c7024db80959247b547f17"}
Oct 30 14:00:02.543862 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:02.543846    5093 scope.go:122] "RemoveContainer" containerID="f73f4c35aacad6329ca64a4a2f639d95cb54b06b6b9848d8ae2e55d0fca3060c"
Oct 30 14:00:02.544169 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:02.543989    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-08c365/test-model-08c365-kserve-576d588df-fsfnh"
Oct 30 14:00:02.616764 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:02.616739    5093 scope.go:122] "RemoveContainer" containerID="f73f4c35aacad6329ca64a4a2f639d95cb54b06b6b9848d8ae2e55d0fca3060c"
Oct 30 14:00:02.617120 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:00:02.617099    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f73f4c35aacad6329ca64a4a2f639d95cb54b06b6b9848d8ae2e55d0fca3060c\": container with ID starting with f73f4c35aacad6329ca64a4a2f639d95cb54b06b6b9848d8ae2e55d0fca3060c not found: ID does not exist" containerID="f73f4c35aacad6329ca64a4a2f639d95cb54b06b6b9848d8ae2e55d0fca3060c"
Oct 30 14:00:02.617188 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:02.617155    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f73f4c35aacad6329ca64a4a2f639d95cb54b06b6b9848d8ae2e55d0fca3060c"} err="failed to get container status \"f73f4c35aacad6329ca64a4a2f639d95cb54b06b6b9848d8ae2e55d0fca3060c\": rpc error: code = NotFound desc = could not find container \"f73f4c35aacad6329ca64a4a2f639d95cb54b06b6b9848d8ae2e55d0fca3060c\": container with ID starting with f73f4c35aacad6329ca64a4a2f639d95cb54b06b6b9848d8ae2e55d0fca3060c not found: ID does not exist"
Oct 30 14:00:02.624835 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:02.624811    5093 scope.go:122] "RemoveContainer" containerID="adcee79ecc2104e6d28b6da391753c52c13369052f24f674b818701afbb12864"
Oct 30 14:00:02.643968 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:02.643803    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889480-r8ddx" podStartSLOduration=2.643785733 podStartE2EDuration="2.643785733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:00:02.640202031 +0000 UTC m=+3421.787469276" watchObservedRunningTime="2026-10-30 14:00:02.643785733 +0000 UTC m=+3421.791052975"
Oct 30 14:00:02.735838 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:02.735775    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889480-hl64w" podStartSLOduration=2.735755065 podStartE2EDuration="2.735755065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:00:02.731827015 +0000 UTC m=+3421.879094257" watchObservedRunningTime="2026-10-30 14:00:02.735755065 +0000 UTC m=+3421.883022307"
Oct 30 14:00:02.787448 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:02.787414    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-08c365/test-model-08c365-kserve-576d588df-fsfnh"]
Oct 30 14:00:02.791790 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:02.791758    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-08c365/test-model-08c365-kserve-576d588df-fsfnh"]
Oct 30 14:00:03.003849 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:03.003811    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f73d682-50c2-46d3-a907-dc4c6fb089af" path="/var/lib/kubelet/pods/2f73d682-50c2-46d3-a907-dc4c6fb089af/volumes"
Oct 30 14:00:03.553994 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:03.553957    5093 generic.go:356] "Generic (PLEG): container finished" podID="dbeb73e2-4d19-404a-9789-b8828f2b2ed4" containerID="bdf8cd413b619c65a718f3be73a4a370464b237b3126da308b49d383da99a88d" exitCode=60
Oct 30 14:00:03.554529 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:03.554049    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889480-r8ddx" event={"ID":"dbeb73e2-4d19-404a-9789-b8828f2b2ed4","Type":"ContainerDied","Data":"bdf8cd413b619c65a718f3be73a4a370464b237b3126da308b49d383da99a88d"}
Oct 30 14:00:03.554529 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:03.554503    5093 scope.go:122] "RemoveContainer" containerID="bdf8cd413b619c65a718f3be73a4a370464b237b3126da308b49d383da99a88d"
Oct 30 14:00:03.556526 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:03.556500    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889480-vwdrd" event={"ID":"992c57a3-547b-4b38-bdcf-139eb477dcbc","Type":"ContainerStarted","Data":"70dfcea74617513b5a90a8d2450138bcc514c61fe5d102585233d8f95e690ecc"}
Oct 30 14:00:03.560992 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:03.560965    5093 generic.go:356] "Generic (PLEG): container finished" podID="41cca4c3-513c-4628-997d-545b0326a9ac" containerID="c8b173302621d139cc605b760f93ff77a40162cd72be1fe7abf645c4c201dcb2" exitCode=60
Oct 30 14:00:03.561142 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:03.561049    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889480-hl64w" event={"ID":"41cca4c3-513c-4628-997d-545b0326a9ac","Type":"ContainerDied","Data":"c8b173302621d139cc605b760f93ff77a40162cd72be1fe7abf645c4c201dcb2"}
Oct 30 14:00:03.561391 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:03.561369    5093 scope.go:122] "RemoveContainer" containerID="c8b173302621d139cc605b760f93ff77a40162cd72be1fe7abf645c4c201dcb2"
Oct 30 14:00:03.563449 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:03.563427    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889480-lxfxn" event={"ID":"073758c5-f33b-459c-a0a7-040c013d39db","Type":"ContainerStarted","Data":"e5ac409f93b8b2af73d6d0e457330f92008d4cbd8e171784bfa0fbb27bb39eab"}
Oct 30 14:00:03.565765 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:03.565739    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889480-qlw8p" event={"ID":"ba607a11-313e-4fd8-b812-771fd8dbbf92","Type":"ContainerStarted","Data":"2b331f43d11090e7a2ae5ac6fac1dd3c6d9fd05e379698cc0addfc150b18c300"}
Oct 30 14:00:03.697067 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:03.696991    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889480-lxfxn" podStartSLOduration=3.696971361 podStartE2EDuration="3.696971361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:00:03.691996407 +0000 UTC m=+3422.839263650" watchObservedRunningTime="2026-10-30 14:00:03.696971361 +0000 UTC m=+3422.844238603"
Oct 30 14:00:03.783701 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:03.783634    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889480-vwdrd" podStartSLOduration=3.783617876 podStartE2EDuration="3.783617876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:00:03.779385802 +0000 UTC m=+3422.926653043" watchObservedRunningTime="2026-10-30 14:00:03.783617876 +0000 UTC m=+3422.930885117"
Oct 30 14:00:04.579916 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:04.579866    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889480-hl64w" event={"ID":"41cca4c3-513c-4628-997d-545b0326a9ac","Type":"ContainerStarted","Data":"2335bdf4261aa2d8ab6a6f6e0d6124d789db43d52f1b2ea868aa783ada042abd"}
Oct 30 14:00:04.582375 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:04.582337    5093 generic.go:356] "Generic (PLEG): container finished" podID="073758c5-f33b-459c-a0a7-040c013d39db" containerID="e5ac409f93b8b2af73d6d0e457330f92008d4cbd8e171784bfa0fbb27bb39eab" exitCode=60
Oct 30 14:00:04.582526 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:04.582432    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889480-lxfxn" event={"ID":"073758c5-f33b-459c-a0a7-040c013d39db","Type":"ContainerDied","Data":"e5ac409f93b8b2af73d6d0e457330f92008d4cbd8e171784bfa0fbb27bb39eab"}
Oct 30 14:00:04.582526 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:04.582507    5093 scope.go:122] "RemoveContainer" containerID="1813bd5a67e909c43499c08c43d9cd58bd2e84c8365eb68b09a547d9b3484e67"
Oct 30 14:00:04.582774 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:04.582750    5093 scope.go:122] "RemoveContainer" containerID="e5ac409f93b8b2af73d6d0e457330f92008d4cbd8e171784bfa0fbb27bb39eab"
Oct 30 14:00:04.583609 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:00:04.583572    5093 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-52af49-29889480-lxfxn_odh-ai-gateway-infra(073758c5-f33b-459c-a0a7-040c013d39db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889480-lxfxn" podUID="073758c5-f33b-459c-a0a7-040c013d39db"
Oct 30 14:00:04.585037 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:04.585016    5093 generic.go:356] "Generic (PLEG): container finished" podID="ba607a11-313e-4fd8-b812-771fd8dbbf92" containerID="2b331f43d11090e7a2ae5ac6fac1dd3c6d9fd05e379698cc0addfc150b18c300" exitCode=60
Oct 30 14:00:04.585182 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:04.585088    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889480-qlw8p" event={"ID":"ba607a11-313e-4fd8-b812-771fd8dbbf92","Type":"ContainerDied","Data":"2b331f43d11090e7a2ae5ac6fac1dd3c6d9fd05e379698cc0addfc150b18c300"}
Oct 30 14:00:04.585408 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:04.585393    5093 scope.go:122] "RemoveContainer" containerID="2b331f43d11090e7a2ae5ac6fac1dd3c6d9fd05e379698cc0addfc150b18c300"
Oct 30 14:00:04.585722 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:00:04.585702    5093 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889480-qlw8p_odh-ai-gateway-infra(ba607a11-313e-4fd8-b812-771fd8dbbf92)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889480-qlw8p" podUID="ba607a11-313e-4fd8-b812-771fd8dbbf92"
Oct 30 14:00:04.587349 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:04.587318    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889480-r8ddx" event={"ID":"dbeb73e2-4d19-404a-9789-b8828f2b2ed4","Type":"ContainerStarted","Data":"95e6463140359874ff86f8f2d55f7dd521dceef4928c270c854858c03039d8e6"}
Oct 30 14:00:04.589959 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:04.589933    5093 generic.go:356] "Generic (PLEG): container finished" podID="992c57a3-547b-4b38-bdcf-139eb477dcbc" containerID="70dfcea74617513b5a90a8d2450138bcc514c61fe5d102585233d8f95e690ecc" exitCode=60
Oct 30 14:00:04.590075 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:04.590000    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889480-vwdrd" event={"ID":"992c57a3-547b-4b38-bdcf-139eb477dcbc","Type":"ContainerDied","Data":"70dfcea74617513b5a90a8d2450138bcc514c61fe5d102585233d8f95e690ecc"}
Oct 30 14:00:04.590375 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:04.590356    5093 scope.go:122] "RemoveContainer" containerID="70dfcea74617513b5a90a8d2450138bcc514c61fe5d102585233d8f95e690ecc"
Oct 30 14:00:04.590711 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:00:04.590684    5093 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889480-vwdrd_odh-ai-gateway-infra(992c57a3-547b-4b38-bdcf-139eb477dcbc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889480-vwdrd" podUID="992c57a3-547b-4b38-bdcf-139eb477dcbc"
Oct 30 14:00:04.603824 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:04.603794    5093 scope.go:122] "RemoveContainer" containerID="9b7fee5a1751a15a9914a659d8a78909edbaf8c534552c79e9a34aa41170044a"
Oct 30 14:00:04.625677 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:04.625643    5093 scope.go:122] "RemoveContainer" containerID="ca3891d5e7c302b08d75aea53765eb5b26f0b1239de6a8da3b7de271a780cb7b"
Oct 30 14:00:05.602977 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:05.602946    5093 scope.go:122] "RemoveContainer" containerID="2b331f43d11090e7a2ae5ac6fac1dd3c6d9fd05e379698cc0addfc150b18c300"
Oct 30 14:00:05.603571 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:00:05.603369    5093 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889480-qlw8p_odh-ai-gateway-infra(ba607a11-313e-4fd8-b812-771fd8dbbf92)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889480-qlw8p" podUID="ba607a11-313e-4fd8-b812-771fd8dbbf92"
Oct 30 14:00:05.604994 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:05.604965    5093 generic.go:356] "Generic (PLEG): container finished" podID="dbeb73e2-4d19-404a-9789-b8828f2b2ed4" containerID="95e6463140359874ff86f8f2d55f7dd521dceef4928c270c854858c03039d8e6" exitCode=60
Oct 30 14:00:05.605201 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:05.605050    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889480-r8ddx" event={"ID":"dbeb73e2-4d19-404a-9789-b8828f2b2ed4","Type":"ContainerDied","Data":"95e6463140359874ff86f8f2d55f7dd521dceef4928c270c854858c03039d8e6"}
Oct 30 14:00:05.605201 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:05.605123    5093 scope.go:122] "RemoveContainer" containerID="bdf8cd413b619c65a718f3be73a4a370464b237b3126da308b49d383da99a88d"
Oct 30 14:00:05.605400 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:05.605378    5093 scope.go:122] "RemoveContainer" containerID="95e6463140359874ff86f8f2d55f7dd521dceef4928c270c854858c03039d8e6"
Oct 30 14:00:05.605808 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:00:05.605782    5093 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889480-r8ddx_odh-ai-gateway-infra(dbeb73e2-4d19-404a-9789-b8828f2b2ed4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889480-r8ddx" podUID="dbeb73e2-4d19-404a-9789-b8828f2b2ed4"
Oct 30 14:00:05.607577 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:05.607548    5093 scope.go:122] "RemoveContainer" containerID="70dfcea74617513b5a90a8d2450138bcc514c61fe5d102585233d8f95e690ecc"
Oct 30 14:00:05.607938 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:00:05.607909    5093 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889480-vwdrd_odh-ai-gateway-infra(992c57a3-547b-4b38-bdcf-139eb477dcbc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889480-vwdrd" podUID="992c57a3-547b-4b38-bdcf-139eb477dcbc"
Oct 30 14:00:05.609268 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:05.609246    5093 generic.go:356] "Generic (PLEG): container finished" podID="41cca4c3-513c-4628-997d-545b0326a9ac" containerID="2335bdf4261aa2d8ab6a6f6e0d6124d789db43d52f1b2ea868aa783ada042abd" exitCode=60
Oct 30 14:00:05.609385 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:05.609339    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889480-hl64w" event={"ID":"41cca4c3-513c-4628-997d-545b0326a9ac","Type":"ContainerDied","Data":"2335bdf4261aa2d8ab6a6f6e0d6124d789db43d52f1b2ea868aa783ada042abd"}
Oct 30 14:00:05.609614 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:05.609597    5093 scope.go:122] "RemoveContainer" containerID="2335bdf4261aa2d8ab6a6f6e0d6124d789db43d52f1b2ea868aa783ada042abd"
Oct 30 14:00:05.609957 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:00:05.609929    5093 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889480-hl64w_odh-ai-gateway-infra(41cca4c3-513c-4628-997d-545b0326a9ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889480-hl64w" podUID="41cca4c3-513c-4628-997d-545b0326a9ac"
Oct 30 14:00:05.611605 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:05.611583    5093 scope.go:122] "RemoveContainer" containerID="e5ac409f93b8b2af73d6d0e457330f92008d4cbd8e171784bfa0fbb27bb39eab"
Oct 30 14:00:05.611979 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:00:05.611954    5093 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-52af49-29889480-lxfxn_odh-ai-gateway-infra(073758c5-f33b-459c-a0a7-040c013d39db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889480-lxfxn" podUID="073758c5-f33b-459c-a0a7-040c013d39db"
Oct 30 14:00:05.620353 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:05.620325    5093 scope.go:122] "RemoveContainer" containerID="c8b173302621d139cc605b760f93ff77a40162cd72be1fe7abf645c4c201dcb2"
Oct 30 14:00:06.622120 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:06.622086    5093 scope.go:122] "RemoveContainer" containerID="95e6463140359874ff86f8f2d55f7dd521dceef4928c270c854858c03039d8e6"
Oct 30 14:00:06.622608 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:00:06.622506    5093 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889480-r8ddx_odh-ai-gateway-infra(dbeb73e2-4d19-404a-9789-b8828f2b2ed4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889480-r8ddx" podUID="dbeb73e2-4d19-404a-9789-b8828f2b2ed4"
Oct 30 14:00:06.623918 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:06.623899    5093 scope.go:122] "RemoveContainer" containerID="2335bdf4261aa2d8ab6a6f6e0d6124d789db43d52f1b2ea868aa783ada042abd"
Oct 30 14:00:06.624242 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:00:06.624224    5093 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889480-hl64w_odh-ai-gateway-infra(41cca4c3-513c-4628-997d-545b0326a9ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889480-hl64w" podUID="41cca4c3-513c-4628-997d-545b0326a9ac"
Oct 30 14:00:14.991523 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:14.991479    5093 scope.go:122] "RemoveContainer" containerID="2b331f43d11090e7a2ae5ac6fac1dd3c6d9fd05e379698cc0addfc150b18c300"
Oct 30 14:00:14.992021 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:14.991602    5093 scope.go:122] "RemoveContainer" containerID="e5ac409f93b8b2af73d6d0e457330f92008d4cbd8e171784bfa0fbb27bb39eab"
Oct 30 14:00:15.713483 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:15.713441    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889480-lxfxn" event={"ID":"073758c5-f33b-459c-a0a7-040c013d39db","Type":"ContainerStarted","Data":"20f39b875aa35e6a0429700cdb829090416de20487e7bd808ec1554cda11735f"}
Oct 30 14:00:15.715753 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:15.715720    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889480-qlw8p" event={"ID":"ba607a11-313e-4fd8-b812-771fd8dbbf92","Type":"ContainerStarted","Data":"1f119e9fe472f95d4ddc0c52ce96c6d2f61452f761678ccfa9f6d99463f6d107"}
Oct 30 14:00:15.992297 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:15.992214    5093 scope.go:122] "RemoveContainer" containerID="95e6463140359874ff86f8f2d55f7dd521dceef4928c270c854858c03039d8e6"
Oct 30 14:00:16.726258 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:16.726214    5093 generic.go:356] "Generic (PLEG): container finished" podID="073758c5-f33b-459c-a0a7-040c013d39db" containerID="20f39b875aa35e6a0429700cdb829090416de20487e7bd808ec1554cda11735f" exitCode=60
Oct 30 14:00:16.726552 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:16.726326    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889480-lxfxn" event={"ID":"073758c5-f33b-459c-a0a7-040c013d39db","Type":"ContainerDied","Data":"20f39b875aa35e6a0429700cdb829090416de20487e7bd808ec1554cda11735f"}
Oct 30 14:00:16.726786 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:16.726759    5093 scope.go:122] "RemoveContainer" containerID="e5ac409f93b8b2af73d6d0e457330f92008d4cbd8e171784bfa0fbb27bb39eab"
Oct 30 14:00:16.727269 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:16.727244    5093 scope.go:122] "RemoveContainer" containerID="20f39b875aa35e6a0429700cdb829090416de20487e7bd808ec1554cda11735f"
Oct 30 14:00:16.727693 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:00:16.727662    5093 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-52af49-29889480-lxfxn_odh-ai-gateway-infra(073758c5-f33b-459c-a0a7-040c013d39db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889480-lxfxn" podUID="073758c5-f33b-459c-a0a7-040c013d39db"
Oct 30 14:00:16.729987 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:16.729959    5093 generic.go:356] "Generic (PLEG): container finished" podID="ba607a11-313e-4fd8-b812-771fd8dbbf92" containerID="1f119e9fe472f95d4ddc0c52ce96c6d2f61452f761678ccfa9f6d99463f6d107" exitCode=60
Oct 30 14:00:16.730121 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:16.730052    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889480-qlw8p" event={"ID":"ba607a11-313e-4fd8-b812-771fd8dbbf92","Type":"ContainerDied","Data":"1f119e9fe472f95d4ddc0c52ce96c6d2f61452f761678ccfa9f6d99463f6d107"}
Oct 30 14:00:16.730600 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:16.730537    5093 scope.go:122] "RemoveContainer" containerID="1f119e9fe472f95d4ddc0c52ce96c6d2f61452f761678ccfa9f6d99463f6d107"
Oct 30 14:00:16.730949 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:00:16.730922    5093 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889480-qlw8p_odh-ai-gateway-infra(ba607a11-313e-4fd8-b812-771fd8dbbf92)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889480-qlw8p" podUID="ba607a11-313e-4fd8-b812-771fd8dbbf92"
Oct 30 14:00:16.732537 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:16.732516    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889480-r8ddx" event={"ID":"dbeb73e2-4d19-404a-9789-b8828f2b2ed4","Type":"ContainerStarted","Data":"75ee61b178f5057bb51d5df6a83176e9e7ae40f539482a364c7fc18fd8a26154"}
Oct 30 14:00:16.744093 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:16.744069    5093 scope.go:122] "RemoveContainer" containerID="2b331f43d11090e7a2ae5ac6fac1dd3c6d9fd05e379698cc0addfc150b18c300"
Oct 30 14:00:16.797563 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:16.797530    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889480-lxfxn"]
Oct 30 14:00:16.850615 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:16.850579    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889480-qlw8p"]
Oct 30 14:00:16.991868 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:16.991763    5093 scope.go:122] "RemoveContainer" containerID="2335bdf4261aa2d8ab6a6f6e0d6124d789db43d52f1b2ea868aa783ada042abd"
Oct 30 14:00:16.992057 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:16.991866    5093 scope.go:122] "RemoveContainer" containerID="70dfcea74617513b5a90a8d2450138bcc514c61fe5d102585233d8f95e690ecc"
Oct 30 14:00:17.742755 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:17.742712    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889480-hl64w" event={"ID":"41cca4c3-513c-4628-997d-545b0326a9ac","Type":"ContainerStarted","Data":"583fadb8ef1fd64ff46934ddad4cfc923b2db80c3be75a8b08f26b6c6dd4b091"}
Oct 30 14:00:17.748899 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:17.748868    5093 generic.go:356] "Generic (PLEG): container finished" podID="dbeb73e2-4d19-404a-9789-b8828f2b2ed4" containerID="75ee61b178f5057bb51d5df6a83176e9e7ae40f539482a364c7fc18fd8a26154" exitCode=60
Oct 30 14:00:17.748998 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:17.748941    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889480-r8ddx" event={"ID":"dbeb73e2-4d19-404a-9789-b8828f2b2ed4","Type":"ContainerDied","Data":"75ee61b178f5057bb51d5df6a83176e9e7ae40f539482a364c7fc18fd8a26154"}
Oct 30 14:00:17.749050 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:17.749020    5093 scope.go:122] "RemoveContainer" containerID="95e6463140359874ff86f8f2d55f7dd521dceef4928c270c854858c03039d8e6"
Oct 30 14:00:17.749539 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:17.749515    5093 scope.go:122] "RemoveContainer" containerID="75ee61b178f5057bb51d5df6a83176e9e7ae40f539482a364c7fc18fd8a26154"
Oct 30 14:00:17.749936 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:00:17.749908    5093 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889480-r8ddx_odh-ai-gateway-infra(dbeb73e2-4d19-404a-9789-b8828f2b2ed4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889480-r8ddx" podUID="dbeb73e2-4d19-404a-9789-b8828f2b2ed4"
Oct 30 14:00:17.751426 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:17.751400    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889480-vwdrd" event={"ID":"992c57a3-547b-4b38-bdcf-139eb477dcbc","Type":"ContainerStarted","Data":"6e6647ec76bdf3ba2eb943b16c09d1b5c53233ba219cb8a9226ff85c9a35abb4"}
Oct 30 14:00:17.926997 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:17.925799    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889480-r8ddx"]
Oct 30 14:00:17.973178 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:17.973091    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889480-lxfxn"
Oct 30 14:00:18.015392 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:18.015321    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889480-qlw8p"
Oct 30 14:00:18.138476 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:18.138443    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba607a11-313e-4fd8-b812-771fd8dbbf92-kube-api-access-46crf\" (UniqueName: \"kubernetes.io/projected/ba607a11-313e-4fd8-b812-771fd8dbbf92-kube-api-access-46crf\") pod \"ba607a11-313e-4fd8-b812-771fd8dbbf92\" (UID: \"ba607a11-313e-4fd8-b812-771fd8dbbf92\") "
Oct 30 14:00:18.138651 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:18.138636    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/073758c5-f33b-459c-a0a7-040c013d39db-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/073758c5-f33b-459c-a0a7-040c013d39db-maas-api-service-ca\") pod \"073758c5-f33b-459c-a0a7-040c013d39db\" (UID: \"073758c5-f33b-459c-a0a7-040c013d39db\") "
Oct 30 14:00:18.138713 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:18.138707    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba607a11-313e-4fd8-b812-771fd8dbbf92-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba607a11-313e-4fd8-b812-771fd8dbbf92-maas-api-service-ca\") pod \"ba607a11-313e-4fd8-b812-771fd8dbbf92\" (UID: \"ba607a11-313e-4fd8-b812-771fd8dbbf92\") "
Oct 30 14:00:18.138836 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:18.138814    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/073758c5-f33b-459c-a0a7-040c013d39db-kube-api-access-l2rjh\" (UniqueName: \"kubernetes.io/projected/073758c5-f33b-459c-a0a7-040c013d39db-kube-api-access-l2rjh\") pod \"073758c5-f33b-459c-a0a7-040c013d39db\" (UID: \"073758c5-f33b-459c-a0a7-040c013d39db\") "
Oct 30 14:00:18.139376 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:18.138979    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/073758c5-f33b-459c-a0a7-040c013d39db-maas-api-service-ca" pod "073758c5-f33b-459c-a0a7-040c013d39db" (UID: "073758c5-f33b-459c-a0a7-040c013d39db"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:00:18.139376 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:18.139066    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba607a11-313e-4fd8-b812-771fd8dbbf92-maas-api-service-ca" pod "ba607a11-313e-4fd8-b812-771fd8dbbf92" (UID: "ba607a11-313e-4fd8-b812-771fd8dbbf92"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:00:18.139584 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:18.139555    5093 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/073758c5-f33b-459c-a0a7-040c013d39db-maas-api-service-ca\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:00:18.139772 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:18.139587    5093 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba607a11-313e-4fd8-b812-771fd8dbbf92-maas-api-service-ca\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:00:18.140712 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:18.140686    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba607a11-313e-4fd8-b812-771fd8dbbf92-kube-api-access-46crf" pod "ba607a11-313e-4fd8-b812-771fd8dbbf92" (UID: "ba607a11-313e-4fd8-b812-771fd8dbbf92"). InnerVolumeSpecName "kube-api-access-46crf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:00:18.141323 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:18.141302    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/073758c5-f33b-459c-a0a7-040c013d39db-kube-api-access-l2rjh" pod "073758c5-f33b-459c-a0a7-040c013d39db" (UID: "073758c5-f33b-459c-a0a7-040c013d39db"). InnerVolumeSpecName "kube-api-access-l2rjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:00:18.241716 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:18.241680    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46crf\" (UniqueName: \"kubernetes.io/projected/ba607a11-313e-4fd8-b812-771fd8dbbf92-kube-api-access-46crf\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:00:18.241716 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:18.241716    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2rjh\" (UniqueName: \"kubernetes.io/projected/073758c5-f33b-459c-a0a7-040c013d39db-kube-api-access-l2rjh\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:00:18.762575 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:18.762539    5093 generic.go:356] "Generic (PLEG): container finished" podID="992c57a3-547b-4b38-bdcf-139eb477dcbc" containerID="6e6647ec76bdf3ba2eb943b16c09d1b5c53233ba219cb8a9226ff85c9a35abb4" exitCode=60
Oct 30 14:00:18.763091 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:18.762615    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889480-vwdrd" event={"ID":"992c57a3-547b-4b38-bdcf-139eb477dcbc","Type":"ContainerDied","Data":"6e6647ec76bdf3ba2eb943b16c09d1b5c53233ba219cb8a9226ff85c9a35abb4"}
Oct 30 14:00:18.763091 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:18.762692    5093 scope.go:122] "RemoveContainer" containerID="70dfcea74617513b5a90a8d2450138bcc514c61fe5d102585233d8f95e690ecc"
Oct 30 14:00:18.763292 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:18.763174    5093 scope.go:122] "RemoveContainer" containerID="6e6647ec76bdf3ba2eb943b16c09d1b5c53233ba219cb8a9226ff85c9a35abb4"
Oct 30 14:00:18.763597 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:00:18.763572    5093 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889480-vwdrd_odh-ai-gateway-infra(992c57a3-547b-4b38-bdcf-139eb477dcbc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889480-vwdrd" podUID="992c57a3-547b-4b38-bdcf-139eb477dcbc"
Oct 30 14:00:18.764954 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:18.764929    5093 generic.go:356] "Generic (PLEG): container finished" podID="41cca4c3-513c-4628-997d-545b0326a9ac" containerID="583fadb8ef1fd64ff46934ddad4cfc923b2db80c3be75a8b08f26b6c6dd4b091" exitCode=60
Oct 30 14:00:18.765072 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:18.764995    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889480-hl64w" event={"ID":"41cca4c3-513c-4628-997d-545b0326a9ac","Type":"ContainerDied","Data":"583fadb8ef1fd64ff46934ddad4cfc923b2db80c3be75a8b08f26b6c6dd4b091"}
Oct 30 14:00:18.765505 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:18.765482    5093 scope.go:122] "RemoveContainer" containerID="583fadb8ef1fd64ff46934ddad4cfc923b2db80c3be75a8b08f26b6c6dd4b091"
Oct 30 14:00:18.765845 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:00:18.765822    5093 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889480-hl64w_odh-ai-gateway-infra(41cca4c3-513c-4628-997d-545b0326a9ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889480-hl64w" podUID="41cca4c3-513c-4628-997d-545b0326a9ac"
Oct 30 14:00:18.766536 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:18.766521    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889480-lxfxn"
Oct 30 14:00:18.766629 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:18.766572    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889480-lxfxn" event={"ID":"073758c5-f33b-459c-a0a7-040c013d39db","Type":"ContainerDied","Data":"c2bb8ac86757c7d353207fc0c2134d901c29bd9799a9e704660ab78974646a46"}
Oct 30 14:00:18.768367 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:18.768341    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889480-qlw8p"
Oct 30 14:00:18.768472 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:18.768430    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889480-qlw8p" event={"ID":"ba607a11-313e-4fd8-b812-771fd8dbbf92","Type":"ContainerDied","Data":"001515d0cb9c7e4fa84e3ffdfc00e0ff2b662791c9f464994a5a4ee16885598b"}
Oct 30 14:00:18.785056 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:18.785029    5093 scope.go:122] "RemoveContainer" containerID="2335bdf4261aa2d8ab6a6f6e0d6124d789db43d52f1b2ea868aa783ada042abd"
Oct 30 14:00:18.803268 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:18.803233    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889480-hl64w"]
Oct 30 14:00:18.804438 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:18.804414    5093 scope.go:122] "RemoveContainer" containerID="20f39b875aa35e6a0429700cdb829090416de20487e7bd808ec1554cda11735f"
Oct 30 14:00:18.821486 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:18.821411    5093 scope.go:122] "RemoveContainer" containerID="1f119e9fe472f95d4ddc0c52ce96c6d2f61452f761678ccfa9f6d99463f6d107"
Oct 30 14:00:18.942198 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:18.942173    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889480-vwdrd"]
Oct 30 14:00:18.973743 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:18.973707    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889480-qlw8p"]
Oct 30 14:00:18.980901 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:18.980875    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889480-r8ddx"
Oct 30 14:00:19.006803 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:19.006771    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c684d2-29889480-qlw8p"]
Oct 30 14:00:19.043303 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:19.043219    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889480-lxfxn"]
Oct 30 14:00:19.049798 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:19.049765    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889480-lxfxn"]
Oct 30 14:00:19.051832 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:19.051766    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbeb73e2-4d19-404a-9789-b8828f2b2ed4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbeb73e2-4d19-404a-9789-b8828f2b2ed4-maas-api-service-ca\") pod \"dbeb73e2-4d19-404a-9789-b8828f2b2ed4\" (UID: \"dbeb73e2-4d19-404a-9789-b8828f2b2ed4\") "
Oct 30 14:00:19.051975 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:19.051960    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbeb73e2-4d19-404a-9789-b8828f2b2ed4-kube-api-access-q2thg\" (UniqueName: \"kubernetes.io/projected/dbeb73e2-4d19-404a-9789-b8828f2b2ed4-kube-api-access-q2thg\") pod \"dbeb73e2-4d19-404a-9789-b8828f2b2ed4\" (UID: \"dbeb73e2-4d19-404a-9789-b8828f2b2ed4\") "
Oct 30 14:00:19.052235 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:19.052210    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbeb73e2-4d19-404a-9789-b8828f2b2ed4-maas-api-service-ca" pod "dbeb73e2-4d19-404a-9789-b8828f2b2ed4" (UID: "dbeb73e2-4d19-404a-9789-b8828f2b2ed4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:00:19.052668 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:19.052644    5093 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbeb73e2-4d19-404a-9789-b8828f2b2ed4-maas-api-service-ca\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:00:19.054294 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:19.054271    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbeb73e2-4d19-404a-9789-b8828f2b2ed4-kube-api-access-q2thg" pod "dbeb73e2-4d19-404a-9789-b8828f2b2ed4" (UID: "dbeb73e2-4d19-404a-9789-b8828f2b2ed4"). InnerVolumeSpecName "kube-api-access-q2thg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:00:19.154567 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:19.154530    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2thg\" (UniqueName: \"kubernetes.io/projected/dbeb73e2-4d19-404a-9789-b8828f2b2ed4-kube-api-access-q2thg\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:00:19.784778 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:19.784742    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889480-r8ddx"
Oct 30 14:00:19.785257 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:19.784773    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889480-r8ddx" event={"ID":"dbeb73e2-4d19-404a-9789-b8828f2b2ed4","Type":"ContainerDied","Data":"21e44d6e104421ef16253942745e92ee7e56f516e68ab403b979bb5515180dab"}
Oct 30 14:00:19.785257 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:19.784880    5093 scope.go:122] "RemoveContainer" containerID="75ee61b178f5057bb51d5df6a83176e9e7ae40f539482a364c7fc18fd8a26154"
Oct 30 14:00:19.906674 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:19.906637    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889480-r8ddx"]
Oct 30 14:00:19.923825 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:19.923779    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-51dc9a-29889480-r8ddx"]
Oct 30 14:00:20.053617 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:20.053531    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889480-hl64w"
Oct 30 14:00:20.058645 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:20.058614    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889480-vwdrd"
Oct 30 14:00:20.072592 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:20.072556    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41cca4c3-513c-4628-997d-545b0326a9ac-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41cca4c3-513c-4628-997d-545b0326a9ac-maas-api-service-ca\") pod \"41cca4c3-513c-4628-997d-545b0326a9ac\" (UID: \"41cca4c3-513c-4628-997d-545b0326a9ac\") "
Oct 30 14:00:20.072954 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:20.072933    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41cca4c3-513c-4628-997d-545b0326a9ac-kube-api-access-ssbbt\" (UniqueName: \"kubernetes.io/projected/41cca4c3-513c-4628-997d-545b0326a9ac-kube-api-access-ssbbt\") pod \"41cca4c3-513c-4628-997d-545b0326a9ac\" (UID: \"41cca4c3-513c-4628-997d-545b0326a9ac\") "
Oct 30 14:00:20.073160 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:20.073125    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/992c57a3-547b-4b38-bdcf-139eb477dcbc-kube-api-access-ncwtz\" (UniqueName: \"kubernetes.io/projected/992c57a3-547b-4b38-bdcf-139eb477dcbc-kube-api-access-ncwtz\") pod \"992c57a3-547b-4b38-bdcf-139eb477dcbc\" (UID: \"992c57a3-547b-4b38-bdcf-139eb477dcbc\") "
Oct 30 14:00:20.074104 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:20.073641    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/992c57a3-547b-4b38-bdcf-139eb477dcbc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/992c57a3-547b-4b38-bdcf-139eb477dcbc-maas-api-service-ca\") pod \"992c57a3-547b-4b38-bdcf-139eb477dcbc\" (UID: \"992c57a3-547b-4b38-bdcf-139eb477dcbc\") "
Oct 30 14:00:20.074344 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:20.072975    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41cca4c3-513c-4628-997d-545b0326a9ac-maas-api-service-ca" pod "41cca4c3-513c-4628-997d-545b0326a9ac" (UID: "41cca4c3-513c-4628-997d-545b0326a9ac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:00:20.074344 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:20.074032    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/992c57a3-547b-4b38-bdcf-139eb477dcbc-maas-api-service-ca" pod "992c57a3-547b-4b38-bdcf-139eb477dcbc" (UID: "992c57a3-547b-4b38-bdcf-139eb477dcbc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:00:20.078787 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:20.078750    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/992c57a3-547b-4b38-bdcf-139eb477dcbc-kube-api-access-ncwtz" pod "992c57a3-547b-4b38-bdcf-139eb477dcbc" (UID: "992c57a3-547b-4b38-bdcf-139eb477dcbc"). InnerVolumeSpecName "kube-api-access-ncwtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:00:20.078787 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:20.078783    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41cca4c3-513c-4628-997d-545b0326a9ac-kube-api-access-ssbbt" pod "41cca4c3-513c-4628-997d-545b0326a9ac" (UID: "41cca4c3-513c-4628-997d-545b0326a9ac"). InnerVolumeSpecName "kube-api-access-ssbbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:00:20.079157 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:20.079110    5093 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/992c57a3-547b-4b38-bdcf-139eb477dcbc-maas-api-service-ca\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:00:20.079280 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:20.079266    5093 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41cca4c3-513c-4628-997d-545b0326a9ac-maas-api-service-ca\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:00:20.180950 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:20.180910    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssbbt\" (UniqueName: \"kubernetes.io/projected/41cca4c3-513c-4628-997d-545b0326a9ac-kube-api-access-ssbbt\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:00:20.180950 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:20.180946    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncwtz\" (UniqueName: \"kubernetes.io/projected/992c57a3-547b-4b38-bdcf-139eb477dcbc-kube-api-access-ncwtz\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:00:20.802863 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:20.802832    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889480-vwdrd"
Oct 30 14:00:20.803342 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:20.802851    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889480-vwdrd" event={"ID":"992c57a3-547b-4b38-bdcf-139eb477dcbc","Type":"ContainerDied","Data":"ba71363ed811257007d2ebc2745b02d2bcb8fb2ee7f2747f0d6ce79b81ffe8c1"}
Oct 30 14:00:20.805594 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:20.803366    5093 scope.go:122] "RemoveContainer" containerID="6e6647ec76bdf3ba2eb943b16c09d1b5c53233ba219cb8a9226ff85c9a35abb4"
Oct 30 14:00:20.805594 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:20.805524    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889480-hl64w" event={"ID":"41cca4c3-513c-4628-997d-545b0326a9ac","Type":"ContainerDied","Data":"94aab6c9c5e74bde4e21561b235901f8410c1c174a63b95faa549b93a941fc25"}
Oct 30 14:00:20.806166 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:20.805641    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889480-hl64w"
Oct 30 14:00:20.819803 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:20.819776    5093 scope.go:122] "RemoveContainer" containerID="583fadb8ef1fd64ff46934ddad4cfc923b2db80c3be75a8b08f26b6c6dd4b091"
Oct 30 14:00:20.876251 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:20.876217    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889480-vwdrd"]
Oct 30 14:00:20.912989 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:20.912948    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889480-vwdrd"]
Oct 30 14:00:20.983418 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:20.983375    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889480-hl64w"]
Oct 30 14:00:21.003017 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:21.002983    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="073758c5-f33b-459c-a0a7-040c013d39db" path="/var/lib/kubelet/pods/073758c5-f33b-459c-a0a7-040c013d39db/volumes"
Oct 30 14:00:21.003509 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:21.003494    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="992c57a3-547b-4b38-bdcf-139eb477dcbc" path="/var/lib/kubelet/pods/992c57a3-547b-4b38-bdcf-139eb477dcbc/volumes"
Oct 30 14:00:21.003905 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:21.003891    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba607a11-313e-4fd8-b812-771fd8dbbf92" path="/var/lib/kubelet/pods/ba607a11-313e-4fd8-b812-771fd8dbbf92/volumes"
Oct 30 14:00:21.004651 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:21.004627    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbeb73e2-4d19-404a-9789-b8828f2b2ed4" path="/var/lib/kubelet/pods/dbeb73e2-4d19-404a-9789-b8828f2b2ed4/volumes"
Oct 30 14:00:21.021821 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:21.021790    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889480-hl64w"]
Oct 30 14:00:21.176038 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:00:21.175997    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 14:00:23.002359 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:23.002317    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41cca4c3-513c-4628-997d-545b0326a9ac" path="/var/lib/kubelet/pods/41cca4c3-513c-4628-997d-545b0326a9ac/volumes"
Oct 30 14:00:24.284585 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:24.284546    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj"]
Oct 30 14:00:24.285097 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:24.284858    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj" podUID="f1b0539d-aead-4e37-b4ba-d6de98ae472b" containerName="istio-proxy" containerID="cri-o://ee3f61d071e8821dca0897b446fc8248dd6d5716db4e988b4ce2727f5d28158e" gracePeriod=30
Oct 30 14:00:25.499858 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:25.499809    5093 patch_prober.go:29] interesting pod/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 14:00:25.500309 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:25.499887    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj" podUID="f1b0539d-aead-4e37-b4ba-d6de98ae472b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 14:00:29.599970 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:29.599939    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj"
Oct 30 14:00:29.760266 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:29.760227    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1b0539d-aead-4e37-b4ba-d6de98ae472b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1b0539d-aead-4e37-b4ba-d6de98ae472b-istiod-ca-cert\") pod \"f1b0539d-aead-4e37-b4ba-d6de98ae472b\" (UID: \"f1b0539d-aead-4e37-b4ba-d6de98ae472b\") "
Oct 30 14:00:29.760465 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:29.760327    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1b0539d-aead-4e37-b4ba-d6de98ae472b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1b0539d-aead-4e37-b4ba-d6de98ae472b-istio-envoy\") pod \"f1b0539d-aead-4e37-b4ba-d6de98ae472b\" (UID: \"f1b0539d-aead-4e37-b4ba-d6de98ae472b\") "
Oct 30 14:00:29.760465 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:29.760374    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1b0539d-aead-4e37-b4ba-d6de98ae472b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1b0539d-aead-4e37-b4ba-d6de98ae472b-istio-data\") pod \"f1b0539d-aead-4e37-b4ba-d6de98ae472b\" (UID: \"f1b0539d-aead-4e37-b4ba-d6de98ae472b\") "
Oct 30 14:00:29.760465 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:29.760412    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1b0539d-aead-4e37-b4ba-d6de98ae472b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1b0539d-aead-4e37-b4ba-d6de98ae472b-istio-podinfo\") pod \"f1b0539d-aead-4e37-b4ba-d6de98ae472b\" (UID: \"f1b0539d-aead-4e37-b4ba-d6de98ae472b\") "
Oct 30 14:00:29.760633 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:29.760547    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1b0539d-aead-4e37-b4ba-d6de98ae472b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1b0539d-aead-4e37-b4ba-d6de98ae472b-workload-socket\") pod \"f1b0539d-aead-4e37-b4ba-d6de98ae472b\" (UID: \"f1b0539d-aead-4e37-b4ba-d6de98ae472b\") "
Oct 30 14:00:29.760633 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:29.760588    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1b0539d-aead-4e37-b4ba-d6de98ae472b-istio-token\" (UniqueName: \"kubernetes.io/projected/f1b0539d-aead-4e37-b4ba-d6de98ae472b-istio-token\") pod \"f1b0539d-aead-4e37-b4ba-d6de98ae472b\" (UID: \"f1b0539d-aead-4e37-b4ba-d6de98ae472b\") "
Oct 30 14:00:29.760633 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:29.760625    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1b0539d-aead-4e37-b4ba-d6de98ae472b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1b0539d-aead-4e37-b4ba-d6de98ae472b-credential-socket\") pod \"f1b0539d-aead-4e37-b4ba-d6de98ae472b\" (UID: \"f1b0539d-aead-4e37-b4ba-d6de98ae472b\") "
Oct 30 14:00:29.760794 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:29.760686    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1b0539d-aead-4e37-b4ba-d6de98ae472b-kube-api-access-6ssf8\" (UniqueName: \"kubernetes.io/projected/f1b0539d-aead-4e37-b4ba-d6de98ae472b-kube-api-access-6ssf8\") pod \"f1b0539d-aead-4e37-b4ba-d6de98ae472b\" (UID: \"f1b0539d-aead-4e37-b4ba-d6de98ae472b\") "
Oct 30 14:00:29.760794 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:29.760722    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1b0539d-aead-4e37-b4ba-d6de98ae472b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1b0539d-aead-4e37-b4ba-d6de98ae472b-workload-certs\") pod \"f1b0539d-aead-4e37-b4ba-d6de98ae472b\" (UID: \"f1b0539d-aead-4e37-b4ba-d6de98ae472b\") "
Oct 30 14:00:29.760794 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:29.760774    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1b0539d-aead-4e37-b4ba-d6de98ae472b-istio-data" pod "f1b0539d-aead-4e37-b4ba-d6de98ae472b" (UID: "f1b0539d-aead-4e37-b4ba-d6de98ae472b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:00:29.760967 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:29.760801    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1b0539d-aead-4e37-b4ba-d6de98ae472b-istiod-ca-cert" pod "f1b0539d-aead-4e37-b4ba-d6de98ae472b" (UID: "f1b0539d-aead-4e37-b4ba-d6de98ae472b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:00:29.761033 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:29.760959    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1b0539d-aead-4e37-b4ba-d6de98ae472b-credential-socket" pod "f1b0539d-aead-4e37-b4ba-d6de98ae472b" (UID: "f1b0539d-aead-4e37-b4ba-d6de98ae472b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:00:29.761097 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:29.761055    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1b0539d-aead-4e37-b4ba-d6de98ae472b-workload-socket" pod "f1b0539d-aead-4e37-b4ba-d6de98ae472b" (UID: "f1b0539d-aead-4e37-b4ba-d6de98ae472b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:00:29.761257 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:29.761236    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1b0539d-aead-4e37-b4ba-d6de98ae472b-workload-certs" pod "f1b0539d-aead-4e37-b4ba-d6de98ae472b" (UID: "f1b0539d-aead-4e37-b4ba-d6de98ae472b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:00:29.761537 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:29.761519    5093 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1b0539d-aead-4e37-b4ba-d6de98ae472b-istiod-ca-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:00:29.761671 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:29.761632    5093 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1b0539d-aead-4e37-b4ba-d6de98ae472b-istio-data\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:00:29.761671 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:29.761653    5093 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1b0539d-aead-4e37-b4ba-d6de98ae472b-workload-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:00:29.761671 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:29.761667    5093 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1b0539d-aead-4e37-b4ba-d6de98ae472b-credential-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:00:29.761911 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:29.761681    5093 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1b0539d-aead-4e37-b4ba-d6de98ae472b-workload-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:00:29.763529 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:29.763503    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1b0539d-aead-4e37-b4ba-d6de98ae472b-istio-envoy" pod "f1b0539d-aead-4e37-b4ba-d6de98ae472b" (UID: "f1b0539d-aead-4e37-b4ba-d6de98ae472b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:00:29.763693 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:29.763561    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1b0539d-aead-4e37-b4ba-d6de98ae472b-kube-api-access-6ssf8" pod "f1b0539d-aead-4e37-b4ba-d6de98ae472b" (UID: "f1b0539d-aead-4e37-b4ba-d6de98ae472b"). InnerVolumeSpecName "kube-api-access-6ssf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:00:29.763693 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:29.763567    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1b0539d-aead-4e37-b4ba-d6de98ae472b-istio-token" pod "f1b0539d-aead-4e37-b4ba-d6de98ae472b" (UID: "f1b0539d-aead-4e37-b4ba-d6de98ae472b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:00:29.763819 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:29.763709    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1b0539d-aead-4e37-b4ba-d6de98ae472b-istio-podinfo" pod "f1b0539d-aead-4e37-b4ba-d6de98ae472b" (UID: "f1b0539d-aead-4e37-b4ba-d6de98ae472b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:00:29.863815 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:29.863700    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1b0539d-aead-4e37-b4ba-d6de98ae472b-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:00:29.863815 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:29.863746    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ssf8\" (UniqueName: \"kubernetes.io/projected/f1b0539d-aead-4e37-b4ba-d6de98ae472b-kube-api-access-6ssf8\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:00:29.863815 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:29.863763    5093 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1b0539d-aead-4e37-b4ba-d6de98ae472b-istio-envoy\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:00:29.863815 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:29.863777    5093 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1b0539d-aead-4e37-b4ba-d6de98ae472b-istio-podinfo\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:00:29.900125 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:29.900085    5093 generic.go:356] "Generic (PLEG): container finished" podID="f1b0539d-aead-4e37-b4ba-d6de98ae472b" containerID="ee3f61d071e8821dca0897b446fc8248dd6d5716db4e988b4ce2727f5d28158e" exitCode=0
Oct 30 14:00:29.900125 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:29.900152    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj"
Oct 30 14:00:29.900421 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:29.900192    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj" event={"ID":"f1b0539d-aead-4e37-b4ba-d6de98ae472b","Type":"ContainerDied","Data":"ee3f61d071e8821dca0897b446fc8248dd6d5716db4e988b4ce2727f5d28158e"}
Oct 30 14:00:29.900421 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:29.900262    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj" event={"ID":"f1b0539d-aead-4e37-b4ba-d6de98ae472b","Type":"ContainerDied","Data":"6e6679b43f256b62b539fb8096db94503c65f60cfc536154764a39e09127b628"}
Oct 30 14:00:29.900421 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:29.900291    5093 scope.go:122] "RemoveContainer" containerID="ee3f61d071e8821dca0897b446fc8248dd6d5716db4e988b4ce2727f5d28158e"
Oct 30 14:00:29.914103 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:29.914078    5093 scope.go:122] "RemoveContainer" containerID="bebfc86760169ac96e7eba6955608f51ee8a4b44910f4ad70af3ac0ae73b004c"
Oct 30 14:00:29.931646 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:29.931616    5093 scope.go:122] "RemoveContainer" containerID="ee3f61d071e8821dca0897b446fc8248dd6d5716db4e988b4ce2727f5d28158e"
Oct 30 14:00:29.932010 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:00:29.931986    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee3f61d071e8821dca0897b446fc8248dd6d5716db4e988b4ce2727f5d28158e\": container with ID starting with ee3f61d071e8821dca0897b446fc8248dd6d5716db4e988b4ce2727f5d28158e not found: ID does not exist" containerID="ee3f61d071e8821dca0897b446fc8248dd6d5716db4e988b4ce2727f5d28158e"
Oct 30 14:00:29.932100 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:29.932024    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee3f61d071e8821dca0897b446fc8248dd6d5716db4e988b4ce2727f5d28158e"} err="failed to get container status \"ee3f61d071e8821dca0897b446fc8248dd6d5716db4e988b4ce2727f5d28158e\": rpc error: code = NotFound desc = could not find container \"ee3f61d071e8821dca0897b446fc8248dd6d5716db4e988b4ce2727f5d28158e\": container with ID starting with ee3f61d071e8821dca0897b446fc8248dd6d5716db4e988b4ce2727f5d28158e not found: ID does not exist"
Oct 30 14:00:29.932100 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:29.932045    5093 scope.go:122] "RemoveContainer" containerID="bebfc86760169ac96e7eba6955608f51ee8a4b44910f4ad70af3ac0ae73b004c"
Oct 30 14:00:29.932403 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:00:29.932381    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bebfc86760169ac96e7eba6955608f51ee8a4b44910f4ad70af3ac0ae73b004c\": container with ID starting with bebfc86760169ac96e7eba6955608f51ee8a4b44910f4ad70af3ac0ae73b004c not found: ID does not exist" containerID="bebfc86760169ac96e7eba6955608f51ee8a4b44910f4ad70af3ac0ae73b004c"
Oct 30 14:00:29.932473 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:29.932407    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bebfc86760169ac96e7eba6955608f51ee8a4b44910f4ad70af3ac0ae73b004c"} err="failed to get container status \"bebfc86760169ac96e7eba6955608f51ee8a4b44910f4ad70af3ac0ae73b004c\": rpc error: code = NotFound desc = could not find container \"bebfc86760169ac96e7eba6955608f51ee8a4b44910f4ad70af3ac0ae73b004c\": container with ID starting with bebfc86760169ac96e7eba6955608f51ee8a4b44910f4ad70af3ac0ae73b004c not found: ID does not exist"
Oct 30 14:00:29.965943 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:29.965904    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj"]
Oct 30 14:00:29.977473 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:29.977425    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-88352d-openshift-default-57cbc595bf-lrsxj"]
Oct 30 14:00:31.002454 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:31.002420    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1b0539d-aead-4e37-b4ba-d6de98ae472b" path="/var/lib/kubelet/pods/f1b0539d-aead-4e37-b4ba-d6de98ae472b/volumes"
Oct 30 14:00:33.766781 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:33.764276    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-926ff0/test-model-926ff0-kserve-84bbd495b5-x2fw2"]
Oct 30 14:00:48.793779 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:48.793673    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-926ff0/test-model-926ff0-kserve-84bbd495b5-x2fw2" podUID="37c1896c-d703-40c3-8be8-94ced507e3e0" containerName="main" containerID="cri-o://efb3c819d4c8b2eb3d758060d673d2f1831f9e69377f68e0ac0d6d681e1b6889" gracePeriod=45
Oct 30 14:00:49.079417 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:49.079313    5093 generic.go:356] "Generic (PLEG): container finished" podID="37c1896c-d703-40c3-8be8-94ced507e3e0" containerID="efb3c819d4c8b2eb3d758060d673d2f1831f9e69377f68e0ac0d6d681e1b6889" exitCode=0
Oct 30 14:00:49.079417 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:49.079386    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-926ff0/test-model-926ff0-kserve-84bbd495b5-x2fw2" event={"ID":"37c1896c-d703-40c3-8be8-94ced507e3e0","Type":"ContainerDied","Data":"efb3c819d4c8b2eb3d758060d673d2f1831f9e69377f68e0ac0d6d681e1b6889"}
Oct 30 14:00:49.139556 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:49.139527    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-926ff0/test-model-926ff0-kserve-84bbd495b5-x2fw2"
Oct 30 14:00:49.247506 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:49.247475    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37c1896c-d703-40c3-8be8-94ced507e3e0-kube-api-access-wgk9q\" (UniqueName: \"kubernetes.io/projected/37c1896c-d703-40c3-8be8-94ced507e3e0-kube-api-access-wgk9q\") pod \"37c1896c-d703-40c3-8be8-94ced507e3e0\" (UID: \"37c1896c-d703-40c3-8be8-94ced507e3e0\") "
Oct 30 14:00:49.247758 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:49.247726    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37c1896c-d703-40c3-8be8-94ced507e3e0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/37c1896c-d703-40c3-8be8-94ced507e3e0-dshm\") pod \"37c1896c-d703-40c3-8be8-94ced507e3e0\" (UID: \"37c1896c-d703-40c3-8be8-94ced507e3e0\") "
Oct 30 14:00:49.247885 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:49.247801    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37c1896c-d703-40c3-8be8-94ced507e3e0-tls-certs\" (UniqueName: \"kubernetes.io/secret/37c1896c-d703-40c3-8be8-94ced507e3e0-tls-certs\") pod \"37c1896c-d703-40c3-8be8-94ced507e3e0\" (UID: \"37c1896c-d703-40c3-8be8-94ced507e3e0\") "
Oct 30 14:00:49.247885 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:49.247869    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37c1896c-d703-40c3-8be8-94ced507e3e0-home\" (UniqueName: \"kubernetes.io/empty-dir/37c1896c-d703-40c3-8be8-94ced507e3e0-home\") pod \"37c1896c-d703-40c3-8be8-94ced507e3e0\" (UID: \"37c1896c-d703-40c3-8be8-94ced507e3e0\") "
Oct 30 14:00:49.248010 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:49.247911    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37c1896c-d703-40c3-8be8-94ced507e3e0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37c1896c-d703-40c3-8be8-94ced507e3e0-tmp-dir\") pod \"37c1896c-d703-40c3-8be8-94ced507e3e0\" (UID: \"37c1896c-d703-40c3-8be8-94ced507e3e0\") "
Oct 30 14:00:49.248076 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:49.248040    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37c1896c-d703-40c3-8be8-94ced507e3e0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37c1896c-d703-40c3-8be8-94ced507e3e0-model-cache\") pod \"37c1896c-d703-40c3-8be8-94ced507e3e0\" (UID: \"37c1896c-d703-40c3-8be8-94ced507e3e0\") "
Oct 30 14:00:49.248625 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:49.248593    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c1896c-d703-40c3-8be8-94ced507e3e0-home" pod "37c1896c-d703-40c3-8be8-94ced507e3e0" (UID: "37c1896c-d703-40c3-8be8-94ced507e3e0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:00:49.248856 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:49.248832    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c1896c-d703-40c3-8be8-94ced507e3e0-tmp-dir" pod "37c1896c-d703-40c3-8be8-94ced507e3e0" (UID: "37c1896c-d703-40c3-8be8-94ced507e3e0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:00:49.249172 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:49.249108    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c1896c-d703-40c3-8be8-94ced507e3e0-model-cache" pod "37c1896c-d703-40c3-8be8-94ced507e3e0" (UID: "37c1896c-d703-40c3-8be8-94ced507e3e0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:00:49.249334 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:49.249311    5093 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37c1896c-d703-40c3-8be8-94ced507e3e0-home\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:00:49.249396 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:49.249341    5093 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37c1896c-d703-40c3-8be8-94ced507e3e0-tmp-dir\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:00:49.249396 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:49.249355    5093 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37c1896c-d703-40c3-8be8-94ced507e3e0-model-cache\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:00:49.250854 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:49.250827    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c1896c-d703-40c3-8be8-94ced507e3e0-dshm" pod "37c1896c-d703-40c3-8be8-94ced507e3e0" (UID: "37c1896c-d703-40c3-8be8-94ced507e3e0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:00:49.251281 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:49.251248    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37c1896c-d703-40c3-8be8-94ced507e3e0-tls-certs" pod "37c1896c-d703-40c3-8be8-94ced507e3e0" (UID: "37c1896c-d703-40c3-8be8-94ced507e3e0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:00:49.251483 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:49.251459    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37c1896c-d703-40c3-8be8-94ced507e3e0-kube-api-access-wgk9q" pod "37c1896c-d703-40c3-8be8-94ced507e3e0" (UID: "37c1896c-d703-40c3-8be8-94ced507e3e0"). InnerVolumeSpecName "kube-api-access-wgk9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:00:49.350965 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:49.350863    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgk9q\" (UniqueName: \"kubernetes.io/projected/37c1896c-d703-40c3-8be8-94ced507e3e0-kube-api-access-wgk9q\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:00:49.350965 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:49.350896    5093 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37c1896c-d703-40c3-8be8-94ced507e3e0-dshm\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:00:49.350965 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:49.350907    5093 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37c1896c-d703-40c3-8be8-94ced507e3e0-tls-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:00:50.089065 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:50.089028    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-926ff0/test-model-926ff0-kserve-84bbd495b5-x2fw2"
Oct 30 14:00:50.089065 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:50.089050    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-926ff0/test-model-926ff0-kserve-84bbd495b5-x2fw2" event={"ID":"37c1896c-d703-40c3-8be8-94ced507e3e0","Type":"ContainerDied","Data":"5dd093c0b98e71cfb4d4f093c1e35079cf13867b2178dc2293e91281ec59675f"}
Oct 30 14:00:50.089630 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:50.089112    5093 scope.go:122] "RemoveContainer" containerID="efb3c819d4c8b2eb3d758060d673d2f1831f9e69377f68e0ac0d6d681e1b6889"
Oct 30 14:00:50.144804 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:50.144767    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-926ff0/test-model-926ff0-kserve-84bbd495b5-x2fw2"]
Oct 30 14:00:50.157729 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:50.157696    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-926ff0/test-model-926ff0-kserve-84bbd495b5-x2fw2"]
Oct 30 14:00:51.003741 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:00:51.003702    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37c1896c-d703-40c3-8be8-94ced507e3e0" path="/var/lib/kubelet/pods/37c1896c-d703-40c3-8be8-94ced507e3e0/volumes"
Oct 30 14:00:51.173388 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:00:51.173342    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 14:01:01.035605 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:01.035564    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2"]
Oct 30 14:01:01.036109 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:01.035852    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2" podUID="abe9a708-c8aa-414b-80a8-fcd5c1142425" containerName="istio-proxy" containerID="cri-o://372e6576dcdda14a024d117f05d63750f8b5e944048018d23df303ef93771a6b" gracePeriod=30
Oct 30 14:01:06.303115 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:06.303025    5093 generic.go:356] "Generic (PLEG): container finished" podID="abe9a708-c8aa-414b-80a8-fcd5c1142425" containerID="372e6576dcdda14a024d117f05d63750f8b5e944048018d23df303ef93771a6b" exitCode=0
Oct 30 14:01:06.303115 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:06.303102    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2" event={"ID":"abe9a708-c8aa-414b-80a8-fcd5c1142425","Type":"ContainerDied","Data":"372e6576dcdda14a024d117f05d63750f8b5e944048018d23df303ef93771a6b"}
Oct 30 14:01:06.303660 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:06.303208    5093 scope.go:122] "RemoveContainer" containerID="6813c626f4ef36b3ac1dedde6e29d5ae2cd0db66fed3061904338d971f29665f"
Oct 30 14:01:06.369368 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:06.369341    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2"
Oct 30 14:01:06.507421 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:06.507388    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abe9a708-c8aa-414b-80a8-fcd5c1142425-kube-api-access-68w8w\" (UniqueName: \"kubernetes.io/projected/abe9a708-c8aa-414b-80a8-fcd5c1142425-kube-api-access-68w8w\") pod \"abe9a708-c8aa-414b-80a8-fcd5c1142425\" (UID: \"abe9a708-c8aa-414b-80a8-fcd5c1142425\") "
Oct 30 14:01:06.507634 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:06.507523    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abe9a708-c8aa-414b-80a8-fcd5c1142425-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abe9a708-c8aa-414b-80a8-fcd5c1142425-istio-envoy\") pod \"abe9a708-c8aa-414b-80a8-fcd5c1142425\" (UID: \"abe9a708-c8aa-414b-80a8-fcd5c1142425\") "
Oct 30 14:01:06.507634 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:06.507544    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abe9a708-c8aa-414b-80a8-fcd5c1142425-istio-token\" (UniqueName: \"kubernetes.io/projected/abe9a708-c8aa-414b-80a8-fcd5c1142425-istio-token\") pod \"abe9a708-c8aa-414b-80a8-fcd5c1142425\" (UID: \"abe9a708-c8aa-414b-80a8-fcd5c1142425\") "
Oct 30 14:01:06.507634 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:06.507571    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abe9a708-c8aa-414b-80a8-fcd5c1142425-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abe9a708-c8aa-414b-80a8-fcd5c1142425-istio-data\") pod \"abe9a708-c8aa-414b-80a8-fcd5c1142425\" (UID: \"abe9a708-c8aa-414b-80a8-fcd5c1142425\") "
Oct 30 14:01:06.507824 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:06.507649    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abe9a708-c8aa-414b-80a8-fcd5c1142425-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe9a708-c8aa-414b-80a8-fcd5c1142425-credential-socket\") pod \"abe9a708-c8aa-414b-80a8-fcd5c1142425\" (UID: \"abe9a708-c8aa-414b-80a8-fcd5c1142425\") "
Oct 30 14:01:06.507824 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:06.507687    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abe9a708-c8aa-414b-80a8-fcd5c1142425-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe9a708-c8aa-414b-80a8-fcd5c1142425-workload-socket\") pod \"abe9a708-c8aa-414b-80a8-fcd5c1142425\" (UID: \"abe9a708-c8aa-414b-80a8-fcd5c1142425\") "
Oct 30 14:01:06.507824 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:06.507739    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/abe9a708-c8aa-414b-80a8-fcd5c1142425-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abe9a708-c8aa-414b-80a8-fcd5c1142425-istio-podinfo\") pod \"abe9a708-c8aa-414b-80a8-fcd5c1142425\" (UID: \"abe9a708-c8aa-414b-80a8-fcd5c1142425\") "
Oct 30 14:01:06.507824 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:06.507770    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abe9a708-c8aa-414b-80a8-fcd5c1142425-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abe9a708-c8aa-414b-80a8-fcd5c1142425-istiod-ca-cert\") pod \"abe9a708-c8aa-414b-80a8-fcd5c1142425\" (UID: \"abe9a708-c8aa-414b-80a8-fcd5c1142425\") "
Oct 30 14:01:06.508045 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:06.507823    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abe9a708-c8aa-414b-80a8-fcd5c1142425-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abe9a708-c8aa-414b-80a8-fcd5c1142425-workload-certs\") pod \"abe9a708-c8aa-414b-80a8-fcd5c1142425\" (UID: \"abe9a708-c8aa-414b-80a8-fcd5c1142425\") "
Oct 30 14:01:06.508045 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:06.507986    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe9a708-c8aa-414b-80a8-fcd5c1142425-istio-data" pod "abe9a708-c8aa-414b-80a8-fcd5c1142425" (UID: "abe9a708-c8aa-414b-80a8-fcd5c1142425"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:01:06.508045 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:06.508002    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe9a708-c8aa-414b-80a8-fcd5c1142425-credential-socket" pod "abe9a708-c8aa-414b-80a8-fcd5c1142425" (UID: "abe9a708-c8aa-414b-80a8-fcd5c1142425"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:01:06.508265 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:06.508054    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe9a708-c8aa-414b-80a8-fcd5c1142425-workload-socket" pod "abe9a708-c8aa-414b-80a8-fcd5c1142425" (UID: "abe9a708-c8aa-414b-80a8-fcd5c1142425"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:01:06.508393 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:06.508334    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe9a708-c8aa-414b-80a8-fcd5c1142425-workload-certs" pod "abe9a708-c8aa-414b-80a8-fcd5c1142425" (UID: "abe9a708-c8aa-414b-80a8-fcd5c1142425"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:01:06.508532 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:06.508501    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abe9a708-c8aa-414b-80a8-fcd5c1142425-istiod-ca-cert" pod "abe9a708-c8aa-414b-80a8-fcd5c1142425" (UID: "abe9a708-c8aa-414b-80a8-fcd5c1142425"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:01:06.508741 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:06.508719    5093 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abe9a708-c8aa-414b-80a8-fcd5c1142425-istio-data\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:01:06.508813 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:06.508750    5093 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe9a708-c8aa-414b-80a8-fcd5c1142425-credential-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:01:06.508813 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:06.508768    5093 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe9a708-c8aa-414b-80a8-fcd5c1142425-workload-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:01:06.508813 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:06.508785    5093 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abe9a708-c8aa-414b-80a8-fcd5c1142425-istiod-ca-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:01:06.508813 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:06.508799    5093 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abe9a708-c8aa-414b-80a8-fcd5c1142425-workload-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:01:06.510289 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:06.510169    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abe9a708-c8aa-414b-80a8-fcd5c1142425-istio-token" pod "abe9a708-c8aa-414b-80a8-fcd5c1142425" (UID: "abe9a708-c8aa-414b-80a8-fcd5c1142425"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:01:06.510410 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:06.510297    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abe9a708-c8aa-414b-80a8-fcd5c1142425-istio-podinfo" pod "abe9a708-c8aa-414b-80a8-fcd5c1142425" (UID: "abe9a708-c8aa-414b-80a8-fcd5c1142425"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:01:06.510410 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:06.510329    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe9a708-c8aa-414b-80a8-fcd5c1142425-istio-envoy" pod "abe9a708-c8aa-414b-80a8-fcd5c1142425" (UID: "abe9a708-c8aa-414b-80a8-fcd5c1142425"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:01:06.510496 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:06.510407    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abe9a708-c8aa-414b-80a8-fcd5c1142425-kube-api-access-68w8w" pod "abe9a708-c8aa-414b-80a8-fcd5c1142425" (UID: "abe9a708-c8aa-414b-80a8-fcd5c1142425"). InnerVolumeSpecName "kube-api-access-68w8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:01:06.609769 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:06.609688    5093 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abe9a708-c8aa-414b-80a8-fcd5c1142425-istio-envoy\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:01:06.609769 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:06.609718    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abe9a708-c8aa-414b-80a8-fcd5c1142425-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:01:06.609769 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:06.609727    5093 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abe9a708-c8aa-414b-80a8-fcd5c1142425-istio-podinfo\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:01:06.609769 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:06.609737    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68w8w\" (UniqueName: \"kubernetes.io/projected/abe9a708-c8aa-414b-80a8-fcd5c1142425-kube-api-access-68w8w\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:01:07.315386 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:07.315310    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2"
Oct 30 14:01:07.315386 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:07.315307    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2" event={"ID":"abe9a708-c8aa-414b-80a8-fcd5c1142425","Type":"ContainerDied","Data":"65ff1d71723cd93f71becc63b06b74ab40c495a535f83440ea3cfc8b67aa3530"}
Oct 30 14:01:07.315883 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:07.315414    5093 scope.go:122] "RemoveContainer" containerID="372e6576dcdda14a024d117f05d63750f8b5e944048018d23df303ef93771a6b"
Oct 30 14:01:07.384213 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:07.384178    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2"]
Oct 30 14:01:07.397382 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:07.397350    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-08c365-openshift-default-7864c56df-k4hr2"]
Oct 30 14:01:09.001991 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:09.001958    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abe9a708-c8aa-414b-80a8-fcd5c1142425" path="/var/lib/kubelet/pods/abe9a708-c8aa-414b-80a8-fcd5c1142425/volumes"
Oct 30 14:01:14.044021 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:14.043981    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj"]
Oct 30 14:01:14.044553 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:14.044247    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj" podUID="9667963f-f8d7-4538-b26f-996d05e67f2f" containerName="istio-proxy" containerID="cri-o://d8c17a1e1bf8dc1540c7875542beba21a928737aad8b85ca36607712907117a5" gracePeriod=30
Oct 30 14:01:18.247638 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:18.247602    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h"]
Oct 30 14:01:18.248110 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:18.247822    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h" podUID="f84ad73e-12f3-4784-9928-bd6a24257682" containerName="istio-proxy" containerID="cri-o://c44220484b3d3c73e15674a80cf7ea18d00b5e59910f94bbcf740e6e0d083c45" gracePeriod=30
Oct 30 14:01:18.261191 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:18.261155    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb"]
Oct 30 14:01:18.261406 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:18.261373    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb" podUID="e97d49cb-0e82-4c03-8aeb-044f1c076234" containerName="istio-proxy" containerID="cri-o://13d4d7fc86791bf2c2b3602f64ed6365c622885be6ac05705b4ca5792d7a8f5c" gracePeriod=30
Oct 30 14:01:18.669099 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:18.669005    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9sgpb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 14:01:18.669099 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:18.669077    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb" podUID="e97d49cb-0e82-4c03-8aeb-044f1c076234" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 14:01:19.338584 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:19.338513    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj"
Oct 30 14:01:19.404884 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:19.404844    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9667963f-f8d7-4538-b26f-996d05e67f2f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9667963f-f8d7-4538-b26f-996d05e67f2f-istio-podinfo\") pod \"9667963f-f8d7-4538-b26f-996d05e67f2f\" (UID: \"9667963f-f8d7-4538-b26f-996d05e67f2f\") "
Oct 30 14:01:19.405069 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:19.404962    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9667963f-f8d7-4538-b26f-996d05e67f2f-kube-api-access-vqxdz\" (UniqueName: \"kubernetes.io/projected/9667963f-f8d7-4538-b26f-996d05e67f2f-kube-api-access-vqxdz\") pod \"9667963f-f8d7-4538-b26f-996d05e67f2f\" (UID: \"9667963f-f8d7-4538-b26f-996d05e67f2f\") "
Oct 30 14:01:19.405069 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:19.404996    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9667963f-f8d7-4538-b26f-996d05e67f2f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9667963f-f8d7-4538-b26f-996d05e67f2f-istiod-ca-cert\") pod \"9667963f-f8d7-4538-b26f-996d05e67f2f\" (UID: \"9667963f-f8d7-4538-b26f-996d05e67f2f\") "
Oct 30 14:01:19.405069 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:19.405049    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9667963f-f8d7-4538-b26f-996d05e67f2f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9667963f-f8d7-4538-b26f-996d05e67f2f-istio-data\") pod \"9667963f-f8d7-4538-b26f-996d05e67f2f\" (UID: \"9667963f-f8d7-4538-b26f-996d05e67f2f\") "
Oct 30 14:01:19.405277 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:19.405124    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9667963f-f8d7-4538-b26f-996d05e67f2f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9667963f-f8d7-4538-b26f-996d05e67f2f-workload-certs\") pod \"9667963f-f8d7-4538-b26f-996d05e67f2f\" (UID: \"9667963f-f8d7-4538-b26f-996d05e67f2f\") "
Oct 30 14:01:19.405277 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:19.405170    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9667963f-f8d7-4538-b26f-996d05e67f2f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9667963f-f8d7-4538-b26f-996d05e67f2f-credential-socket\") pod \"9667963f-f8d7-4538-b26f-996d05e67f2f\" (UID: \"9667963f-f8d7-4538-b26f-996d05e67f2f\") "
Oct 30 14:01:19.405277 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:19.405218    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9667963f-f8d7-4538-b26f-996d05e67f2f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9667963f-f8d7-4538-b26f-996d05e67f2f-workload-socket\") pod \"9667963f-f8d7-4538-b26f-996d05e67f2f\" (UID: \"9667963f-f8d7-4538-b26f-996d05e67f2f\") "
Oct 30 14:01:19.405451 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:19.405302    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9667963f-f8d7-4538-b26f-996d05e67f2f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9667963f-f8d7-4538-b26f-996d05e67f2f-istio-envoy\") pod \"9667963f-f8d7-4538-b26f-996d05e67f2f\" (UID: \"9667963f-f8d7-4538-b26f-996d05e67f2f\") "
Oct 30 14:01:19.405451 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:19.405395    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9667963f-f8d7-4538-b26f-996d05e67f2f-istio-token\" (UniqueName: \"kubernetes.io/projected/9667963f-f8d7-4538-b26f-996d05e67f2f-istio-token\") pod \"9667963f-f8d7-4538-b26f-996d05e67f2f\" (UID: \"9667963f-f8d7-4538-b26f-996d05e67f2f\") "
Oct 30 14:01:19.405579 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:19.405547    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9667963f-f8d7-4538-b26f-996d05e67f2f-workload-certs" pod "9667963f-f8d7-4538-b26f-996d05e67f2f" (UID: "9667963f-f8d7-4538-b26f-996d05e67f2f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:01:19.405805 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:19.405782    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9667963f-f8d7-4538-b26f-996d05e67f2f-istio-data" pod "9667963f-f8d7-4538-b26f-996d05e67f2f" (UID: "9667963f-f8d7-4538-b26f-996d05e67f2f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:01:19.405907 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:19.405822    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9667963f-f8d7-4538-b26f-996d05e67f2f-istiod-ca-cert" pod "9667963f-f8d7-4538-b26f-996d05e67f2f" (UID: "9667963f-f8d7-4538-b26f-996d05e67f2f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:01:19.406019 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:19.405998    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9667963f-f8d7-4538-b26f-996d05e67f2f-credential-socket" pod "9667963f-f8d7-4538-b26f-996d05e67f2f" (UID: "9667963f-f8d7-4538-b26f-996d05e67f2f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:01:19.406334 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:19.406216    5093 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9667963f-f8d7-4538-b26f-996d05e67f2f-istiod-ca-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:01:19.406334 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:19.406242    5093 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9667963f-f8d7-4538-b26f-996d05e67f2f-istio-data\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:01:19.406334 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:19.406255    5093 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9667963f-f8d7-4538-b26f-996d05e67f2f-workload-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:01:19.406334 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:19.406268    5093 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9667963f-f8d7-4538-b26f-996d05e67f2f-credential-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:01:19.406734 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:19.406685    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9667963f-f8d7-4538-b26f-996d05e67f2f-workload-socket" pod "9667963f-f8d7-4538-b26f-996d05e67f2f" (UID: "9667963f-f8d7-4538-b26f-996d05e67f2f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:01:19.408272 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:19.408226    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9667963f-f8d7-4538-b26f-996d05e67f2f-istio-token" pod "9667963f-f8d7-4538-b26f-996d05e67f2f" (UID: "9667963f-f8d7-4538-b26f-996d05e67f2f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:01:19.408991 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:19.408959    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9667963f-f8d7-4538-b26f-996d05e67f2f-kube-api-access-vqxdz" pod "9667963f-f8d7-4538-b26f-996d05e67f2f" (UID: "9667963f-f8d7-4538-b26f-996d05e67f2f"). InnerVolumeSpecName "kube-api-access-vqxdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:01:19.411155 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:19.411100    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9667963f-f8d7-4538-b26f-996d05e67f2f-istio-podinfo" pod "9667963f-f8d7-4538-b26f-996d05e67f2f" (UID: "9667963f-f8d7-4538-b26f-996d05e67f2f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:01:19.414680 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:19.414653    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9667963f-f8d7-4538-b26f-996d05e67f2f-istio-envoy" pod "9667963f-f8d7-4538-b26f-996d05e67f2f" (UID: "9667963f-f8d7-4538-b26f-996d05e67f2f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:01:19.418539 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:19.418506    5093 generic.go:356] "Generic (PLEG): container finished" podID="9667963f-f8d7-4538-b26f-996d05e67f2f" containerID="d8c17a1e1bf8dc1540c7875542beba21a928737aad8b85ca36607712907117a5" exitCode=0
Oct 30 14:01:19.418633 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:19.418554    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj"
Oct 30 14:01:19.418633 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:19.418561    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj" event={"ID":"9667963f-f8d7-4538-b26f-996d05e67f2f","Type":"ContainerDied","Data":"d8c17a1e1bf8dc1540c7875542beba21a928737aad8b85ca36607712907117a5"}
Oct 30 14:01:19.418633 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:19.418626    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj" event={"ID":"9667963f-f8d7-4538-b26f-996d05e67f2f","Type":"ContainerDied","Data":"65d19f47ac33ad4027d83c9c04fa7f22c5d33b8341e23f393be870a682c075c2"}
Oct 30 14:01:19.418758 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:19.418657    5093 scope.go:122] "RemoveContainer" containerID="d8c17a1e1bf8dc1540c7875542beba21a928737aad8b85ca36607712907117a5"
Oct 30 14:01:19.454560 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:19.454537    5093 scope.go:122] "RemoveContainer" containerID="cfbdfbb0cf6eccafe1e81e745fccdcac947a8c84e9c1af56c409f5bcb387cec7"
Oct 30 14:01:19.465493 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:19.465467    5093 scope.go:122] "RemoveContainer" containerID="d8c17a1e1bf8dc1540c7875542beba21a928737aad8b85ca36607712907117a5"
Oct 30 14:01:19.465822 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:01:19.465801    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8c17a1e1bf8dc1540c7875542beba21a928737aad8b85ca36607712907117a5\": container with ID starting with d8c17a1e1bf8dc1540c7875542beba21a928737aad8b85ca36607712907117a5 not found: ID does not exist" containerID="d8c17a1e1bf8dc1540c7875542beba21a928737aad8b85ca36607712907117a5"
Oct 30 14:01:19.465906 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:19.465834    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8c17a1e1bf8dc1540c7875542beba21a928737aad8b85ca36607712907117a5"} err="failed to get container status \"d8c17a1e1bf8dc1540c7875542beba21a928737aad8b85ca36607712907117a5\": rpc error: code = NotFound desc = could not find container \"d8c17a1e1bf8dc1540c7875542beba21a928737aad8b85ca36607712907117a5\": container with ID starting with d8c17a1e1bf8dc1540c7875542beba21a928737aad8b85ca36607712907117a5 not found: ID does not exist"
Oct 30 14:01:19.465906 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:19.465853    5093 scope.go:122] "RemoveContainer" containerID="cfbdfbb0cf6eccafe1e81e745fccdcac947a8c84e9c1af56c409f5bcb387cec7"
Oct 30 14:01:19.466192 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:01:19.466172    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfbdfbb0cf6eccafe1e81e745fccdcac947a8c84e9c1af56c409f5bcb387cec7\": container with ID starting with cfbdfbb0cf6eccafe1e81e745fccdcac947a8c84e9c1af56c409f5bcb387cec7 not found: ID does not exist" containerID="cfbdfbb0cf6eccafe1e81e745fccdcac947a8c84e9c1af56c409f5bcb387cec7"
Oct 30 14:01:19.466268 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:19.466197    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfbdfbb0cf6eccafe1e81e745fccdcac947a8c84e9c1af56c409f5bcb387cec7"} err="failed to get container status \"cfbdfbb0cf6eccafe1e81e745fccdcac947a8c84e9c1af56c409f5bcb387cec7\": rpc error: code = NotFound desc = could not find container \"cfbdfbb0cf6eccafe1e81e745fccdcac947a8c84e9c1af56c409f5bcb387cec7\": container with ID starting with cfbdfbb0cf6eccafe1e81e745fccdcac947a8c84e9c1af56c409f5bcb387cec7 not found: ID does not exist"
Oct 30 14:01:19.492360 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:19.492326    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj"]
Oct 30 14:01:19.496786 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:19.496752    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-blkhj"]
Oct 30 14:01:19.507860 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:19.507832    5093 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9667963f-f8d7-4538-b26f-996d05e67f2f-istio-podinfo\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:01:19.507860 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:19.507862    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqxdz\" (UniqueName: \"kubernetes.io/projected/9667963f-f8d7-4538-b26f-996d05e67f2f-kube-api-access-vqxdz\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:01:19.508189 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:19.507873    5093 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9667963f-f8d7-4538-b26f-996d05e67f2f-workload-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:01:19.508189 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:19.507883    5093 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9667963f-f8d7-4538-b26f-996d05e67f2f-istio-envoy\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:01:19.508189 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:19.507891    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9667963f-f8d7-4538-b26f-996d05e67f2f-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:01:21.000948 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:21.000909    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9667963f-f8d7-4538-b26f-996d05e67f2f" path="/var/lib/kubelet/pods/9667963f-f8d7-4538-b26f-996d05e67f2f/volumes"
Oct 30 14:01:21.170198 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:01:21.170151    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 14:01:23.489669 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.489632    5093 generic.go:356] "Generic (PLEG): container finished" podID="f84ad73e-12f3-4784-9928-bd6a24257682" containerID="c44220484b3d3c73e15674a80cf7ea18d00b5e59910f94bbcf740e6e0d083c45" exitCode=0
Oct 30 14:01:23.490347 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.490316    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h" event={"ID":"f84ad73e-12f3-4784-9928-bd6a24257682","Type":"ContainerDied","Data":"c44220484b3d3c73e15674a80cf7ea18d00b5e59910f94bbcf740e6e0d083c45"}
Oct 30 14:01:23.499949 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.499496    5093 generic.go:356] "Generic (PLEG): container finished" podID="e97d49cb-0e82-4c03-8aeb-044f1c076234" containerID="13d4d7fc86791bf2c2b3602f64ed6365c622885be6ac05705b4ca5792d7a8f5c" exitCode=0
Oct 30 14:01:23.499949 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.499727    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb" event={"ID":"e97d49cb-0e82-4c03-8aeb-044f1c076234","Type":"ContainerDied","Data":"13d4d7fc86791bf2c2b3602f64ed6365c622885be6ac05705b4ca5792d7a8f5c"}
Oct 30 14:01:23.615254 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.615216    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h"
Oct 30 14:01:23.655906 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.655874    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb"
Oct 30 14:01:23.780159 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.780103    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e97d49cb-0e82-4c03-8aeb-044f1c076234-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e97d49cb-0e82-4c03-8aeb-044f1c076234-istio-envoy\") pod \"e97d49cb-0e82-4c03-8aeb-044f1c076234\" (UID: \"e97d49cb-0e82-4c03-8aeb-044f1c076234\") "
Oct 30 14:01:23.780369 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.780203    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f84ad73e-12f3-4784-9928-bd6a24257682-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f84ad73e-12f3-4784-9928-bd6a24257682-istio-data\") pod \"f84ad73e-12f3-4784-9928-bd6a24257682\" (UID: \"f84ad73e-12f3-4784-9928-bd6a24257682\") "
Oct 30 14:01:23.780369 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.780256    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f84ad73e-12f3-4784-9928-bd6a24257682-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f84ad73e-12f3-4784-9928-bd6a24257682-istio-envoy\") pod \"f84ad73e-12f3-4784-9928-bd6a24257682\" (UID: \"f84ad73e-12f3-4784-9928-bd6a24257682\") "
Oct 30 14:01:23.780369 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.780307    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f84ad73e-12f3-4784-9928-bd6a24257682-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f84ad73e-12f3-4784-9928-bd6a24257682-istio-podinfo\") pod \"f84ad73e-12f3-4784-9928-bd6a24257682\" (UID: \"f84ad73e-12f3-4784-9928-bd6a24257682\") "
Oct 30 14:01:23.780369 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.780356    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f84ad73e-12f3-4784-9928-bd6a24257682-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84ad73e-12f3-4784-9928-bd6a24257682-credential-socket\") pod \"f84ad73e-12f3-4784-9928-bd6a24257682\" (UID: \"f84ad73e-12f3-4784-9928-bd6a24257682\") "
Oct 30 14:01:23.780612 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.780400    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e97d49cb-0e82-4c03-8aeb-044f1c076234-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e97d49cb-0e82-4c03-8aeb-044f1c076234-workload-certs\") pod \"e97d49cb-0e82-4c03-8aeb-044f1c076234\" (UID: \"e97d49cb-0e82-4c03-8aeb-044f1c076234\") "
Oct 30 14:01:23.780612 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.780432    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f84ad73e-12f3-4784-9928-bd6a24257682-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84ad73e-12f3-4784-9928-bd6a24257682-workload-socket\") pod \"f84ad73e-12f3-4784-9928-bd6a24257682\" (UID: \"f84ad73e-12f3-4784-9928-bd6a24257682\") "
Oct 30 14:01:23.780612 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.780462    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e97d49cb-0e82-4c03-8aeb-044f1c076234-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e97d49cb-0e82-4c03-8aeb-044f1c076234-credential-socket\") pod \"e97d49cb-0e82-4c03-8aeb-044f1c076234\" (UID: \"e97d49cb-0e82-4c03-8aeb-044f1c076234\") "
Oct 30 14:01:23.780612 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.780551    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f84ad73e-12f3-4784-9928-bd6a24257682-istio-token\" (UniqueName: \"kubernetes.io/projected/f84ad73e-12f3-4784-9928-bd6a24257682-istio-token\") pod \"f84ad73e-12f3-4784-9928-bd6a24257682\" (UID: \"f84ad73e-12f3-4784-9928-bd6a24257682\") "
Oct 30 14:01:23.780612 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.780586    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e97d49cb-0e82-4c03-8aeb-044f1c076234-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e97d49cb-0e82-4c03-8aeb-044f1c076234-istio-data\") pod \"e97d49cb-0e82-4c03-8aeb-044f1c076234\" (UID: \"e97d49cb-0e82-4c03-8aeb-044f1c076234\") "
Oct 30 14:01:23.780883 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.780616    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e97d49cb-0e82-4c03-8aeb-044f1c076234-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e97d49cb-0e82-4c03-8aeb-044f1c076234-workload-socket\") pod \"e97d49cb-0e82-4c03-8aeb-044f1c076234\" (UID: \"e97d49cb-0e82-4c03-8aeb-044f1c076234\") "
Oct 30 14:01:23.780883 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.780685    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e97d49cb-0e82-4c03-8aeb-044f1c076234-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e97d49cb-0e82-4c03-8aeb-044f1c076234-istio-podinfo\") pod \"e97d49cb-0e82-4c03-8aeb-044f1c076234\" (UID: \"e97d49cb-0e82-4c03-8aeb-044f1c076234\") "
Oct 30 14:01:23.780883 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.780772    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e97d49cb-0e82-4c03-8aeb-044f1c076234-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e97d49cb-0e82-4c03-8aeb-044f1c076234-istiod-ca-cert\") pod \"e97d49cb-0e82-4c03-8aeb-044f1c076234\" (UID: \"e97d49cb-0e82-4c03-8aeb-044f1c076234\") "
Oct 30 14:01:23.780883 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.780785    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f84ad73e-12f3-4784-9928-bd6a24257682-credential-socket" pod "f84ad73e-12f3-4784-9928-bd6a24257682" (UID: "f84ad73e-12f3-4784-9928-bd6a24257682"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:01:23.780883 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.780818    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f84ad73e-12f3-4784-9928-bd6a24257682-kube-api-access-p8v8l\" (UniqueName: \"kubernetes.io/projected/f84ad73e-12f3-4784-9928-bd6a24257682-kube-api-access-p8v8l\") pod \"f84ad73e-12f3-4784-9928-bd6a24257682\" (UID: \"f84ad73e-12f3-4784-9928-bd6a24257682\") "
Oct 30 14:01:23.780883 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.780856    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f84ad73e-12f3-4784-9928-bd6a24257682-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f84ad73e-12f3-4784-9928-bd6a24257682-workload-certs\") pod \"f84ad73e-12f3-4784-9928-bd6a24257682\" (UID: \"f84ad73e-12f3-4784-9928-bd6a24257682\") "
Oct 30 14:01:23.781240 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.780969    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e97d49cb-0e82-4c03-8aeb-044f1c076234-kube-api-access-qj4vw\" (UniqueName: \"kubernetes.io/projected/e97d49cb-0e82-4c03-8aeb-044f1c076234-kube-api-access-qj4vw\") pod \"e97d49cb-0e82-4c03-8aeb-044f1c076234\" (UID: \"e97d49cb-0e82-4c03-8aeb-044f1c076234\") "
Oct 30 14:01:23.781240 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.780995    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f84ad73e-12f3-4784-9928-bd6a24257682-istio-data" pod "f84ad73e-12f3-4784-9928-bd6a24257682" (UID: "f84ad73e-12f3-4784-9928-bd6a24257682"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:01:23.781240 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.781029    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e97d49cb-0e82-4c03-8aeb-044f1c076234-istio-token\" (UniqueName: \"kubernetes.io/projected/e97d49cb-0e82-4c03-8aeb-044f1c076234-istio-token\") pod \"e97d49cb-0e82-4c03-8aeb-044f1c076234\" (UID: \"e97d49cb-0e82-4c03-8aeb-044f1c076234\") "
Oct 30 14:01:23.781240 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.781058    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f84ad73e-12f3-4784-9928-bd6a24257682-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f84ad73e-12f3-4784-9928-bd6a24257682-istiod-ca-cert\") pod \"f84ad73e-12f3-4784-9928-bd6a24257682\" (UID: \"f84ad73e-12f3-4784-9928-bd6a24257682\") "
Oct 30 14:01:23.782776 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.781716    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e97d49cb-0e82-4c03-8aeb-044f1c076234-workload-socket" pod "e97d49cb-0e82-4c03-8aeb-044f1c076234" (UID: "e97d49cb-0e82-4c03-8aeb-044f1c076234"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:01:23.782776 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.781722    5093 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f84ad73e-12f3-4784-9928-bd6a24257682-istio-data\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:01:23.782776 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.781765    5093 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84ad73e-12f3-4784-9928-bd6a24257682-credential-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:01:23.782776 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.781995    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e97d49cb-0e82-4c03-8aeb-044f1c076234-workload-certs" pod "e97d49cb-0e82-4c03-8aeb-044f1c076234" (UID: "e97d49cb-0e82-4c03-8aeb-044f1c076234"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:01:23.782776 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.782073    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f84ad73e-12f3-4784-9928-bd6a24257682-istiod-ca-cert" pod "f84ad73e-12f3-4784-9928-bd6a24257682" (UID: "f84ad73e-12f3-4784-9928-bd6a24257682"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:01:23.782776 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.782231    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f84ad73e-12f3-4784-9928-bd6a24257682-workload-socket" pod "f84ad73e-12f3-4784-9928-bd6a24257682" (UID: "f84ad73e-12f3-4784-9928-bd6a24257682"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:01:23.782776 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.782431    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e97d49cb-0e82-4c03-8aeb-044f1c076234-credential-socket" pod "e97d49cb-0e82-4c03-8aeb-044f1c076234" (UID: "e97d49cb-0e82-4c03-8aeb-044f1c076234"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:01:23.784278 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.782990    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e97d49cb-0e82-4c03-8aeb-044f1c076234-istio-envoy" pod "e97d49cb-0e82-4c03-8aeb-044f1c076234" (UID: "e97d49cb-0e82-4c03-8aeb-044f1c076234"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:01:23.784278 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.783065    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f84ad73e-12f3-4784-9928-bd6a24257682-workload-certs" pod "f84ad73e-12f3-4784-9928-bd6a24257682" (UID: "f84ad73e-12f3-4784-9928-bd6a24257682"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:01:23.784278 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.783403    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e97d49cb-0e82-4c03-8aeb-044f1c076234-istiod-ca-cert" pod "e97d49cb-0e82-4c03-8aeb-044f1c076234" (UID: "e97d49cb-0e82-4c03-8aeb-044f1c076234"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:01:23.784278 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.783636    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e97d49cb-0e82-4c03-8aeb-044f1c076234-istio-data" pod "e97d49cb-0e82-4c03-8aeb-044f1c076234" (UID: "e97d49cb-0e82-4c03-8aeb-044f1c076234"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:01:23.785926 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.785895    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e97d49cb-0e82-4c03-8aeb-044f1c076234-kube-api-access-qj4vw" pod "e97d49cb-0e82-4c03-8aeb-044f1c076234" (UID: "e97d49cb-0e82-4c03-8aeb-044f1c076234"). InnerVolumeSpecName "kube-api-access-qj4vw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:01:23.787946 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.787873    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e97d49cb-0e82-4c03-8aeb-044f1c076234-istio-podinfo" pod "e97d49cb-0e82-4c03-8aeb-044f1c076234" (UID: "e97d49cb-0e82-4c03-8aeb-044f1c076234"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:01:23.787946 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.787900    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e97d49cb-0e82-4c03-8aeb-044f1c076234-istio-token" pod "e97d49cb-0e82-4c03-8aeb-044f1c076234" (UID: "e97d49cb-0e82-4c03-8aeb-044f1c076234"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:01:23.788725 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.788679    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f84ad73e-12f3-4784-9928-bd6a24257682-kube-api-access-p8v8l" pod "f84ad73e-12f3-4784-9928-bd6a24257682" (UID: "f84ad73e-12f3-4784-9928-bd6a24257682"). InnerVolumeSpecName "kube-api-access-p8v8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:01:23.789156 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.789113    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f84ad73e-12f3-4784-9928-bd6a24257682-istio-token" pod "f84ad73e-12f3-4784-9928-bd6a24257682" (UID: "f84ad73e-12f3-4784-9928-bd6a24257682"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:01:23.789396 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.789372    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f84ad73e-12f3-4784-9928-bd6a24257682-istio-envoy" pod "f84ad73e-12f3-4784-9928-bd6a24257682" (UID: "f84ad73e-12f3-4784-9928-bd6a24257682"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:01:23.789652 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.789630    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f84ad73e-12f3-4784-9928-bd6a24257682-istio-podinfo" pod "f84ad73e-12f3-4784-9928-bd6a24257682" (UID: "f84ad73e-12f3-4784-9928-bd6a24257682"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:01:23.883024 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.882937    5093 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e97d49cb-0e82-4c03-8aeb-044f1c076234-istiod-ca-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:01:23.883024 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.882970    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8v8l\" (UniqueName: \"kubernetes.io/projected/f84ad73e-12f3-4784-9928-bd6a24257682-kube-api-access-p8v8l\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:01:23.883024 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.882980    5093 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f84ad73e-12f3-4784-9928-bd6a24257682-workload-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:01:23.883024 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.882989    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj4vw\" (UniqueName: \"kubernetes.io/projected/e97d49cb-0e82-4c03-8aeb-044f1c076234-kube-api-access-qj4vw\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:01:23.883024 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.882998    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e97d49cb-0e82-4c03-8aeb-044f1c076234-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:01:23.883024 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.883011    5093 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f84ad73e-12f3-4784-9928-bd6a24257682-istiod-ca-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:01:23.883024 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.883019    5093 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e97d49cb-0e82-4c03-8aeb-044f1c076234-istio-envoy\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:01:23.883024 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.883027    5093 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f84ad73e-12f3-4784-9928-bd6a24257682-istio-envoy\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:01:23.883473 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.883035    5093 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f84ad73e-12f3-4784-9928-bd6a24257682-istio-podinfo\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:01:23.883473 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.883043    5093 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e97d49cb-0e82-4c03-8aeb-044f1c076234-workload-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:01:23.883473 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.883052    5093 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84ad73e-12f3-4784-9928-bd6a24257682-workload-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:01:23.883473 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.883059    5093 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e97d49cb-0e82-4c03-8aeb-044f1c076234-credential-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:01:23.883473 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.883068    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f84ad73e-12f3-4784-9928-bd6a24257682-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:01:23.883473 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.883076    5093 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e97d49cb-0e82-4c03-8aeb-044f1c076234-istio-data\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:01:23.883473 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.883086    5093 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e97d49cb-0e82-4c03-8aeb-044f1c076234-workload-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:01:23.883473 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:23.883094    5093 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e97d49cb-0e82-4c03-8aeb-044f1c076234-istio-podinfo\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:01:24.508648 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:24.508611    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb"
Oct 30 14:01:24.508648 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:24.508631    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb" event={"ID":"e97d49cb-0e82-4c03-8aeb-044f1c076234","Type":"ContainerDied","Data":"2905bde47151f8e64fe21de23057cc2a086db9e488f3a428f0602057b5eea29e"}
Oct 30 14:01:24.509207 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:24.508701    5093 scope.go:122] "RemoveContainer" containerID="13d4d7fc86791bf2c2b3602f64ed6365c622885be6ac05705b4ca5792d7a8f5c"
Oct 30 14:01:24.510376 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:24.510354    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h" event={"ID":"f84ad73e-12f3-4784-9928-bd6a24257682","Type":"ContainerDied","Data":"e53f233a7e7db7f392ed32be8991401f59487563ceab9de1e7792db75c68c2e8"}
Oct 30 14:01:24.510376 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:24.510368    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h"
Oct 30 14:01:24.524108 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:24.524056    5093 scope.go:122] "RemoveContainer" containerID="c44220484b3d3c73e15674a80cf7ea18d00b5e59910f94bbcf740e6e0d083c45"
Oct 30 14:01:24.578200 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:24.578095    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb"]
Oct 30 14:01:24.591218 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:24.591187    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9sgpb"]
Oct 30 14:01:24.629162 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:24.629094    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h"]
Oct 30 14:01:24.636602 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:24.636561    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7xk7h"]
Oct 30 14:01:25.000806 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:25.000767    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e97d49cb-0e82-4c03-8aeb-044f1c076234" path="/var/lib/kubelet/pods/e97d49cb-0e82-4c03-8aeb-044f1c076234/volumes"
Oct 30 14:01:25.001313 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:25.001297    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f84ad73e-12f3-4784-9928-bd6a24257682" path="/var/lib/kubelet/pods/f84ad73e-12f3-4784-9928-bd6a24257682/volumes"
Oct 30 14:01:45.163314 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:45.160406    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-926ff0-6b69d56f7f-cbkkk"]
Oct 30 14:01:45.163314 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:45.160756    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-926ff0-6b69d56f7f-cbkkk" podUID="80042047-5b26-4633-98f6-d0ee98d70d11" containerName="maas-api" containerID="cri-o://2fd1128bba46f78007ecca3375dd90105b3aff8e94885e43648af5ff748e3a12" gracePeriod=30
Oct 30 14:01:45.472034 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:45.472004    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-926ff0-6b69d56f7f-cbkkk"
Oct 30 14:01:45.584562 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:45.584525    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80042047-5b26-4633-98f6-d0ee98d70d11-metrics-certs\" (UniqueName: \"kubernetes.io/secret/80042047-5b26-4633-98f6-d0ee98d70d11-metrics-certs\") pod \"80042047-5b26-4633-98f6-d0ee98d70d11\" (UID: \"80042047-5b26-4633-98f6-d0ee98d70d11\") "
Oct 30 14:01:45.584745 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:45.584649    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80042047-5b26-4633-98f6-d0ee98d70d11-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80042047-5b26-4633-98f6-d0ee98d70d11-openshift-service-ca\") pod \"80042047-5b26-4633-98f6-d0ee98d70d11\" (UID: \"80042047-5b26-4633-98f6-d0ee98d70d11\") "
Oct 30 14:01:45.584745 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:45.584677    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80042047-5b26-4633-98f6-d0ee98d70d11-kube-api-access-2zk44\" (UniqueName: \"kubernetes.io/projected/80042047-5b26-4633-98f6-d0ee98d70d11-kube-api-access-2zk44\") pod \"80042047-5b26-4633-98f6-d0ee98d70d11\" (UID: \"80042047-5b26-4633-98f6-d0ee98d70d11\") "
Oct 30 14:01:45.584745 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:45.584708    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80042047-5b26-4633-98f6-d0ee98d70d11-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80042047-5b26-4633-98f6-d0ee98d70d11-maas-api-tls\") pod \"80042047-5b26-4633-98f6-d0ee98d70d11\" (UID: \"80042047-5b26-4633-98f6-d0ee98d70d11\") "
Oct 30 14:01:45.585041 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:45.585008    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80042047-5b26-4633-98f6-d0ee98d70d11-openshift-service-ca" pod "80042047-5b26-4633-98f6-d0ee98d70d11" (UID: "80042047-5b26-4633-98f6-d0ee98d70d11"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:01:45.585372 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:45.585342    5093 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80042047-5b26-4633-98f6-d0ee98d70d11-openshift-service-ca\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:01:45.586842 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:45.586814    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80042047-5b26-4633-98f6-d0ee98d70d11-metrics-certs" pod "80042047-5b26-4633-98f6-d0ee98d70d11" (UID: "80042047-5b26-4633-98f6-d0ee98d70d11"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:01:45.587403 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:45.587378    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80042047-5b26-4633-98f6-d0ee98d70d11-maas-api-tls" pod "80042047-5b26-4633-98f6-d0ee98d70d11" (UID: "80042047-5b26-4633-98f6-d0ee98d70d11"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:01:45.587508 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:45.587422    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80042047-5b26-4633-98f6-d0ee98d70d11-kube-api-access-2zk44" pod "80042047-5b26-4633-98f6-d0ee98d70d11" (UID: "80042047-5b26-4633-98f6-d0ee98d70d11"). InnerVolumeSpecName "kube-api-access-2zk44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:01:45.686622 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:45.686524    5093 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80042047-5b26-4633-98f6-d0ee98d70d11-maas-api-tls\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:01:45.686622 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:45.686559    5093 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80042047-5b26-4633-98f6-d0ee98d70d11-metrics-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:01:45.686622 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:45.686569    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zk44\" (UniqueName: \"kubernetes.io/projected/80042047-5b26-4633-98f6-d0ee98d70d11-kube-api-access-2zk44\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:01:45.716001 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:45.715960    5093 generic.go:356] "Generic (PLEG): container finished" podID="80042047-5b26-4633-98f6-d0ee98d70d11" containerID="2fd1128bba46f78007ecca3375dd90105b3aff8e94885e43648af5ff748e3a12" exitCode=0
Oct 30 14:01:45.716211 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:45.716005    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-926ff0-6b69d56f7f-cbkkk"
Oct 30 14:01:45.716211 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:45.716042    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-926ff0-6b69d56f7f-cbkkk" event={"ID":"80042047-5b26-4633-98f6-d0ee98d70d11","Type":"ContainerDied","Data":"2fd1128bba46f78007ecca3375dd90105b3aff8e94885e43648af5ff748e3a12"}
Oct 30 14:01:45.716211 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:45.716101    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-926ff0-6b69d56f7f-cbkkk" event={"ID":"80042047-5b26-4633-98f6-d0ee98d70d11","Type":"ContainerDied","Data":"a90136063588e94f9675cde3aa6f8105cd717ac9e49ceb2c09a714e9b427493e"}
Oct 30 14:01:45.716211 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:45.716161    5093 scope.go:122] "RemoveContainer" containerID="2fd1128bba46f78007ecca3375dd90105b3aff8e94885e43648af5ff748e3a12"
Oct 30 14:01:45.729048 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:45.729023    5093 scope.go:122] "RemoveContainer" containerID="2fd1128bba46f78007ecca3375dd90105b3aff8e94885e43648af5ff748e3a12"
Oct 30 14:01:45.729386 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:01:45.729355    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fd1128bba46f78007ecca3375dd90105b3aff8e94885e43648af5ff748e3a12\": container with ID starting with 2fd1128bba46f78007ecca3375dd90105b3aff8e94885e43648af5ff748e3a12 not found: ID does not exist" containerID="2fd1128bba46f78007ecca3375dd90105b3aff8e94885e43648af5ff748e3a12"
Oct 30 14:01:45.729486 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:45.729397    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fd1128bba46f78007ecca3375dd90105b3aff8e94885e43648af5ff748e3a12"} err="failed to get container status \"2fd1128bba46f78007ecca3375dd90105b3aff8e94885e43648af5ff748e3a12\": rpc error: code = NotFound desc = could not find container \"2fd1128bba46f78007ecca3375dd90105b3aff8e94885e43648af5ff748e3a12\": container with ID starting with 2fd1128bba46f78007ecca3375dd90105b3aff8e94885e43648af5ff748e3a12 not found: ID does not exist"
Oct 30 14:01:45.775753 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:45.775707    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-926ff0-6b69d56f7f-cbkkk"]
Oct 30 14:01:45.795043 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:45.795006    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-926ff0-6b69d56f7f-cbkkk"]
Oct 30 14:01:47.000534 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:47.000497    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80042047-5b26-4633-98f6-d0ee98d70d11" path="/var/lib/kubelet/pods/80042047-5b26-4633-98f6-d0ee98d70d11/volumes"
Oct 30 14:01:51.176249 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:01:51.176205    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 14:01:52.488079 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:52.488040    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-09014b-7d8bb679ff-msw5h"]
Oct 30 14:01:52.488722 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:52.488370    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09014b-7d8bb679ff-msw5h" podUID="19dc1706-24ed-4126-8ee7-b35fd6fe9d37" containerName="maas-api" containerID="cri-o://5b0d9611fcf4ba7797041366cbd099031221f2c2b63cd4a1009acc896df1f0e0" gracePeriod=30
Oct 30 14:01:52.777815 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:52.777729    5093 generic.go:356] "Generic (PLEG): container finished" podID="19dc1706-24ed-4126-8ee7-b35fd6fe9d37" containerID="5b0d9611fcf4ba7797041366cbd099031221f2c2b63cd4a1009acc896df1f0e0" exitCode=0
Oct 30 14:01:52.777815 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:52.777799    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09014b-7d8bb679ff-msw5h" event={"ID":"19dc1706-24ed-4126-8ee7-b35fd6fe9d37","Type":"ContainerDied","Data":"5b0d9611fcf4ba7797041366cbd099031221f2c2b63cd4a1009acc896df1f0e0"}
Oct 30 14:01:52.777919 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:52.777853    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09014b-7d8bb679ff-msw5h" event={"ID":"19dc1706-24ed-4126-8ee7-b35fd6fe9d37","Type":"ContainerDied","Data":"6a33f177bfefb51f7a3db270b3b5511a1988c5e6dab6485cb8860b5aa93ae54d"}
Oct 30 14:01:52.777919 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:52.777872    5093 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a33f177bfefb51f7a3db270b3b5511a1988c5e6dab6485cb8860b5aa93ae54d"
Oct 30 14:01:52.804369 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:52.804340    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09014b-7d8bb679ff-msw5h"
Oct 30 14:01:52.916820 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:52.916776    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19dc1706-24ed-4126-8ee7-b35fd6fe9d37-metrics-certs\" (UniqueName: \"kubernetes.io/secret/19dc1706-24ed-4126-8ee7-b35fd6fe9d37-metrics-certs\") pod \"19dc1706-24ed-4126-8ee7-b35fd6fe9d37\" (UID: \"19dc1706-24ed-4126-8ee7-b35fd6fe9d37\") "
Oct 30 14:01:52.917026 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:52.916843    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19dc1706-24ed-4126-8ee7-b35fd6fe9d37-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19dc1706-24ed-4126-8ee7-b35fd6fe9d37-openshift-service-ca\") pod \"19dc1706-24ed-4126-8ee7-b35fd6fe9d37\" (UID: \"19dc1706-24ed-4126-8ee7-b35fd6fe9d37\") "
Oct 30 14:01:52.917106 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:52.917066    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19dc1706-24ed-4126-8ee7-b35fd6fe9d37-kube-api-access-g8r6x\" (UniqueName: \"kubernetes.io/projected/19dc1706-24ed-4126-8ee7-b35fd6fe9d37-kube-api-access-g8r6x\") pod \"19dc1706-24ed-4126-8ee7-b35fd6fe9d37\" (UID: \"19dc1706-24ed-4126-8ee7-b35fd6fe9d37\") "
Oct 30 14:01:52.917222 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:52.917100    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19dc1706-24ed-4126-8ee7-b35fd6fe9d37-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19dc1706-24ed-4126-8ee7-b35fd6fe9d37-maas-api-tls\") pod \"19dc1706-24ed-4126-8ee7-b35fd6fe9d37\" (UID: \"19dc1706-24ed-4126-8ee7-b35fd6fe9d37\") "
Oct 30 14:01:52.917341 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:52.917313    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19dc1706-24ed-4126-8ee7-b35fd6fe9d37-openshift-service-ca" pod "19dc1706-24ed-4126-8ee7-b35fd6fe9d37" (UID: "19dc1706-24ed-4126-8ee7-b35fd6fe9d37"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:01:52.917733 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:52.917713    5093 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19dc1706-24ed-4126-8ee7-b35fd6fe9d37-openshift-service-ca\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:01:52.919272 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:52.919245    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19dc1706-24ed-4126-8ee7-b35fd6fe9d37-maas-api-tls" pod "19dc1706-24ed-4126-8ee7-b35fd6fe9d37" (UID: "19dc1706-24ed-4126-8ee7-b35fd6fe9d37"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:01:52.919367 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:52.919340    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19dc1706-24ed-4126-8ee7-b35fd6fe9d37-kube-api-access-g8r6x" pod "19dc1706-24ed-4126-8ee7-b35fd6fe9d37" (UID: "19dc1706-24ed-4126-8ee7-b35fd6fe9d37"). InnerVolumeSpecName "kube-api-access-g8r6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:01:52.919487 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:52.919467    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19dc1706-24ed-4126-8ee7-b35fd6fe9d37-metrics-certs" pod "19dc1706-24ed-4126-8ee7-b35fd6fe9d37" (UID: "19dc1706-24ed-4126-8ee7-b35fd6fe9d37"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:01:53.018935 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:53.018897    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8r6x\" (UniqueName: \"kubernetes.io/projected/19dc1706-24ed-4126-8ee7-b35fd6fe9d37-kube-api-access-g8r6x\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:01:53.018935 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:53.018937    5093 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19dc1706-24ed-4126-8ee7-b35fd6fe9d37-maas-api-tls\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:01:53.019275 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:53.018952    5093 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19dc1706-24ed-4126-8ee7-b35fd6fe9d37-metrics-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:01:53.785510 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:53.785472    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09014b-7d8bb679ff-msw5h"
Oct 30 14:01:53.842165 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:53.842114    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-09014b-7d8bb679ff-msw5h"]
Oct 30 14:01:53.854276 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:53.854234    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-09014b-7d8bb679ff-msw5h"]
Oct 30 14:01:55.010492 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:01:55.010455    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19dc1706-24ed-4126-8ee7-b35fd6fe9d37" path="/var/lib/kubelet/pods/19dc1706-24ed-4126-8ee7-b35fd6fe9d37/volumes"
Oct 30 14:02:02.833055 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:02.833015    5093 scope.go:122] "RemoveContainer" containerID="5b0d9611fcf4ba7797041366cbd099031221f2c2b63cd4a1009acc896df1f0e0"
Oct 30 14:02:03.406810 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.406772    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5"]
Oct 30 14:02:03.407523 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407506    5093 cpu_manager.go:414] "containerMap: removing container" podUID="41cca4c3-513c-4628-997d-545b0326a9ac" containerName="cleanup"
Oct 30 14:02:03.407595 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407524    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41cca4c3-513c-4628-997d-545b0326a9ac" containerName="cleanup"
Oct 30 14:02:03.407595 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407535    5093 cpu_manager.go:414] "containerMap: removing container" podUID="41cca4c3-513c-4628-997d-545b0326a9ac" containerName="cleanup"
Oct 30 14:02:03.407595 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407540    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41cca4c3-513c-4628-997d-545b0326a9ac" containerName="cleanup"
Oct 30 14:02:03.407595 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407551    5093 cpu_manager.go:414] "containerMap: removing container" podUID="f84ad73e-12f3-4784-9928-bd6a24257682" containerName="istio-proxy"
Oct 30 14:02:03.407595 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407556    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f84ad73e-12f3-4784-9928-bd6a24257682" containerName="istio-proxy"
Oct 30 14:02:03.407595 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407565    5093 cpu_manager.go:414] "containerMap: removing container" podUID="f1b0539d-aead-4e37-b4ba-d6de98ae472b" containerName="istio-proxy"
Oct 30 14:02:03.407595 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407570    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1b0539d-aead-4e37-b4ba-d6de98ae472b" containerName="istio-proxy"
Oct 30 14:02:03.407595 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407576    5093 cpu_manager.go:414] "containerMap: removing container" podUID="992c57a3-547b-4b38-bdcf-139eb477dcbc" containerName="cleanup"
Oct 30 14:02:03.407595 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407580    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="992c57a3-547b-4b38-bdcf-139eb477dcbc" containerName="cleanup"
Oct 30 14:02:03.407595 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407588    5093 cpu_manager.go:414] "containerMap: removing container" podUID="9667963f-f8d7-4538-b26f-996d05e67f2f" containerName="istio-proxy"
Oct 30 14:02:03.407595 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407594    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9667963f-f8d7-4538-b26f-996d05e67f2f" containerName="istio-proxy"
Oct 30 14:02:03.407595 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407600    5093 cpu_manager.go:414] "containerMap: removing container" podUID="ba607a11-313e-4fd8-b812-771fd8dbbf92" containerName="cleanup"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407605    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba607a11-313e-4fd8-b812-771fd8dbbf92" containerName="cleanup"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407613    5093 cpu_manager.go:414] "containerMap: removing container" podUID="ba607a11-313e-4fd8-b812-771fd8dbbf92" containerName="cleanup"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407617    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba607a11-313e-4fd8-b812-771fd8dbbf92" containerName="cleanup"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407624    5093 cpu_manager.go:414] "containerMap: removing container" podUID="992c57a3-547b-4b38-bdcf-139eb477dcbc" containerName="cleanup"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407628    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="992c57a3-547b-4b38-bdcf-139eb477dcbc" containerName="cleanup"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407636    5093 cpu_manager.go:414] "containerMap: removing container" podUID="9667963f-f8d7-4538-b26f-996d05e67f2f" containerName="istio-proxy"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407640    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9667963f-f8d7-4538-b26f-996d05e67f2f" containerName="istio-proxy"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407648    5093 cpu_manager.go:414] "containerMap: removing container" podUID="37c1896c-d703-40c3-8be8-94ced507e3e0" containerName="main"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407652    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37c1896c-d703-40c3-8be8-94ced507e3e0" containerName="main"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407665    5093 cpu_manager.go:414] "containerMap: removing container" podUID="abe9a708-c8aa-414b-80a8-fcd5c1142425" containerName="istio-proxy"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407670    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abe9a708-c8aa-414b-80a8-fcd5c1142425" containerName="istio-proxy"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407679    5093 cpu_manager.go:414] "containerMap: removing container" podUID="abe9a708-c8aa-414b-80a8-fcd5c1142425" containerName="istio-proxy"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407683    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abe9a708-c8aa-414b-80a8-fcd5c1142425" containerName="istio-proxy"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407689    5093 cpu_manager.go:414] "containerMap: removing container" podUID="073758c5-f33b-459c-a0a7-040c013d39db" containerName="cleanup"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407694    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="073758c5-f33b-459c-a0a7-040c013d39db" containerName="cleanup"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407699    5093 cpu_manager.go:414] "containerMap: removing container" podUID="e97d49cb-0e82-4c03-8aeb-044f1c076234" containerName="istio-proxy"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407703    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e97d49cb-0e82-4c03-8aeb-044f1c076234" containerName="istio-proxy"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407717    5093 cpu_manager.go:414] "containerMap: removing container" podUID="073758c5-f33b-459c-a0a7-040c013d39db" containerName="cleanup"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407721    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="073758c5-f33b-459c-a0a7-040c013d39db" containerName="cleanup"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407734    5093 cpu_manager.go:414] "containerMap: removing container" podUID="80042047-5b26-4633-98f6-d0ee98d70d11" containerName="maas-api"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407739    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80042047-5b26-4633-98f6-d0ee98d70d11" containerName="maas-api"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407746    5093 cpu_manager.go:414] "containerMap: removing container" podUID="2f73d682-50c2-46d3-a907-dc4c6fb089af" containerName="main"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407750    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f73d682-50c2-46d3-a907-dc4c6fb089af" containerName="main"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407756    5093 cpu_manager.go:414] "containerMap: removing container" podUID="dbeb73e2-4d19-404a-9789-b8828f2b2ed4" containerName="cleanup"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407760    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbeb73e2-4d19-404a-9789-b8828f2b2ed4" containerName="cleanup"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407773    5093 cpu_manager.go:414] "containerMap: removing container" podUID="19dc1706-24ed-4126-8ee7-b35fd6fe9d37" containerName="maas-api"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407777    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19dc1706-24ed-4126-8ee7-b35fd6fe9d37" containerName="maas-api"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407784    5093 cpu_manager.go:414] "containerMap: removing container" podUID="dbeb73e2-4d19-404a-9789-b8828f2b2ed4" containerName="cleanup"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407789    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbeb73e2-4d19-404a-9789-b8828f2b2ed4" containerName="cleanup"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407803    5093 cpu_manager.go:414] "containerMap: removing container" podUID="073758c5-f33b-459c-a0a7-040c013d39db" containerName="cleanup"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407808    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="073758c5-f33b-459c-a0a7-040c013d39db" containerName="cleanup"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407817    5093 cpu_manager.go:414] "containerMap: removing container" podUID="dbeb73e2-4d19-404a-9789-b8828f2b2ed4" containerName="cleanup"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407821    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbeb73e2-4d19-404a-9789-b8828f2b2ed4" containerName="cleanup"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407910    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="abe9a708-c8aa-414b-80a8-fcd5c1142425" containerName="istio-proxy"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407920    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="abe9a708-c8aa-414b-80a8-fcd5c1142425" containerName="istio-proxy"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407926    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1b0539d-aead-4e37-b4ba-d6de98ae472b" containerName="istio-proxy"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407932    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="073758c5-f33b-459c-a0a7-040c013d39db" containerName="cleanup"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407937    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="992c57a3-547b-4b38-bdcf-139eb477dcbc" containerName="cleanup"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407943    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="e97d49cb-0e82-4c03-8aeb-044f1c076234" containerName="istio-proxy"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407950    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba607a11-313e-4fd8-b812-771fd8dbbf92" containerName="cleanup"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407956    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="073758c5-f33b-459c-a0a7-040c013d39db" containerName="cleanup"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407962    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="80042047-5b26-4633-98f6-d0ee98d70d11" containerName="maas-api"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407968    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f73d682-50c2-46d3-a907-dc4c6fb089af" containerName="main"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407973    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbeb73e2-4d19-404a-9789-b8828f2b2ed4" containerName="cleanup"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407978    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="41cca4c3-513c-4628-997d-545b0326a9ac" containerName="cleanup"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407985    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="19dc1706-24ed-4126-8ee7-b35fd6fe9d37" containerName="maas-api"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407990    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbeb73e2-4d19-404a-9789-b8828f2b2ed4" containerName="cleanup"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.407998    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="41cca4c3-513c-4628-997d-545b0326a9ac" containerName="cleanup"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.408003    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="41cca4c3-513c-4628-997d-545b0326a9ac" containerName="cleanup"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.408009    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="f84ad73e-12f3-4784-9928-bd6a24257682" containerName="istio-proxy"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.408014    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="9667963f-f8d7-4538-b26f-996d05e67f2f" containerName="istio-proxy"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.408019    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba607a11-313e-4fd8-b812-771fd8dbbf92" containerName="cleanup"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.408025    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="992c57a3-547b-4b38-bdcf-139eb477dcbc" containerName="cleanup"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.408031    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="9667963f-f8d7-4538-b26f-996d05e67f2f" containerName="istio-proxy"
Oct 30 14:02:03.408229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.408037    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="37c1896c-d703-40c3-8be8-94ced507e3e0" containerName="main"
Oct 30 14:02:03.413839 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.413812    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5"
Oct 30 14:02:03.425761 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.425735    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b"]
Oct 30 14:02:03.427686 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.427660    5093 cpu_manager.go:414] "containerMap: removing container" podUID="41cca4c3-513c-4628-997d-545b0326a9ac" containerName="cleanup"
Oct 30 14:02:03.427768 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.427689    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41cca4c3-513c-4628-997d-545b0326a9ac" containerName="cleanup"
Oct 30 14:02:03.427768 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.427706    5093 cpu_manager.go:414] "containerMap: removing container" podUID="f1b0539d-aead-4e37-b4ba-d6de98ae472b" containerName="istio-proxy"
Oct 30 14:02:03.427768 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.427714    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1b0539d-aead-4e37-b4ba-d6de98ae472b" containerName="istio-proxy"
Oct 30 14:02:03.427768 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.427729    5093 cpu_manager.go:414] "containerMap: removing container" podUID="ba607a11-313e-4fd8-b812-771fd8dbbf92" containerName="cleanup"
Oct 30 14:02:03.427768 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.427737    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba607a11-313e-4fd8-b812-771fd8dbbf92" containerName="cleanup"
Oct 30 14:02:03.427768 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.427747    5093 cpu_manager.go:414] "containerMap: removing container" podUID="992c57a3-547b-4b38-bdcf-139eb477dcbc" containerName="cleanup"
Oct 30 14:02:03.427768 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.427754    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="992c57a3-547b-4b38-bdcf-139eb477dcbc" containerName="cleanup"
Oct 30 14:02:03.428029 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.428015    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1b0539d-aead-4e37-b4ba-d6de98ae472b" containerName="istio-proxy"
Oct 30 14:02:03.428069 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.428033    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="992c57a3-547b-4b38-bdcf-139eb477dcbc" containerName="cleanup"
Oct 30 14:02:03.428122 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.428071    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba607a11-313e-4fd8-b812-771fd8dbbf92" containerName="cleanup"
Oct 30 14:02:03.428122 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.428083    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="073758c5-f33b-459c-a0a7-040c013d39db" containerName="cleanup"
Oct 30 14:02:03.428122 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.428098    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbeb73e2-4d19-404a-9789-b8828f2b2ed4" containerName="cleanup"
Oct 30 14:02:03.432492 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.432470    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b"
Oct 30 14:02:03.434932 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.434904    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-cw6t5\""
Oct 30 14:02:03.440922 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.440894    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5"]
Oct 30 14:02:03.458573 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.458539    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b"]
Oct 30 14:02:03.476829 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.476789    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnpl7\" (UniqueName: \"kubernetes.io/projected/f9778494-18c5-4eb6-926b-92f69cbde312-kube-api-access-cnpl7\") pod \"maas-default-gateway-openshift-default-67bf986f99-b6ph5\" (UID: \"f9778494-18c5-4eb6-926b-92f69cbde312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5"
Oct 30 14:02:03.476829 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.476844    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sspzw\" (UniqueName: \"kubernetes.io/projected/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-kube-api-access-sspzw\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2k8b\" (UID: \"762bb1a6-9339-4c49-bf97-1e160e7b0fc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b"
Oct 30 14:02:03.477220 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.476922    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2k8b\" (UID: \"762bb1a6-9339-4c49-bf97-1e160e7b0fc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b"
Oct 30 14:02:03.477220 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.476968    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2k8b\" (UID: \"762bb1a6-9339-4c49-bf97-1e160e7b0fc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b"
Oct 30 14:02:03.477220 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.477033    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9778494-18c5-4eb6-926b-92f69cbde312-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-b6ph5\" (UID: \"f9778494-18c5-4eb6-926b-92f69cbde312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5"
Oct 30 14:02:03.477220 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.477057    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2k8b\" (UID: \"762bb1a6-9339-4c49-bf97-1e160e7b0fc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b"
Oct 30 14:02:03.477220 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.477085    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9778494-18c5-4eb6-926b-92f69cbde312-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-b6ph5\" (UID: \"f9778494-18c5-4eb6-926b-92f69cbde312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5"
Oct 30 14:02:03.477220 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.477112    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2k8b\" (UID: \"762bb1a6-9339-4c49-bf97-1e160e7b0fc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b"
Oct 30 14:02:03.477220 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.477198    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9778494-18c5-4eb6-926b-92f69cbde312-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-b6ph5\" (UID: \"f9778494-18c5-4eb6-926b-92f69cbde312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5"
Oct 30 14:02:03.477657 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.477323    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9778494-18c5-4eb6-926b-92f69cbde312-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-b6ph5\" (UID: \"f9778494-18c5-4eb6-926b-92f69cbde312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5"
Oct 30 14:02:03.477657 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.477489    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9778494-18c5-4eb6-926b-92f69cbde312-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-b6ph5\" (UID: \"f9778494-18c5-4eb6-926b-92f69cbde312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5"
Oct 30 14:02:03.477657 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.477545    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9778494-18c5-4eb6-926b-92f69cbde312-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-b6ph5\" (UID: \"f9778494-18c5-4eb6-926b-92f69cbde312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5"
Oct 30 14:02:03.477657 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.477582    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2k8b\" (UID: \"762bb1a6-9339-4c49-bf97-1e160e7b0fc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b"
Oct 30 14:02:03.477657 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.477608    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2k8b\" (UID: \"762bb1a6-9339-4c49-bf97-1e160e7b0fc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b"
Oct 30 14:02:03.477657 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.477645    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2k8b\" (UID: \"762bb1a6-9339-4c49-bf97-1e160e7b0fc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b"
Oct 30 14:02:03.477992 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.477677    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9778494-18c5-4eb6-926b-92f69cbde312-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-b6ph5\" (UID: \"f9778494-18c5-4eb6-926b-92f69cbde312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5"
Oct 30 14:02:03.477992 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.477709    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9778494-18c5-4eb6-926b-92f69cbde312-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-b6ph5\" (UID: \"f9778494-18c5-4eb6-926b-92f69cbde312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5"
Oct 30 14:02:03.477992 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.477741    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2k8b\" (UID: \"762bb1a6-9339-4c49-bf97-1e160e7b0fc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b"
Oct 30 14:02:03.578649 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.578614    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2k8b\" (UID: \"762bb1a6-9339-4c49-bf97-1e160e7b0fc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b"
Oct 30 14:02:03.578863 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.578663    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2k8b\" (UID: \"762bb1a6-9339-4c49-bf97-1e160e7b0fc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b"
Oct 30 14:02:03.578863 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.578702    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9778494-18c5-4eb6-926b-92f69cbde312-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-b6ph5\" (UID: \"f9778494-18c5-4eb6-926b-92f69cbde312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5"
Oct 30 14:02:03.578863 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.578719    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2k8b\" (UID: \"762bb1a6-9339-4c49-bf97-1e160e7b0fc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b"
Oct 30 14:02:03.578863 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.578737    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9778494-18c5-4eb6-926b-92f69cbde312-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-b6ph5\" (UID: \"f9778494-18c5-4eb6-926b-92f69cbde312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5"
Oct 30 14:02:03.578863 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.578757    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2k8b\" (UID: \"762bb1a6-9339-4c49-bf97-1e160e7b0fc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b"
Oct 30 14:02:03.578863 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.578815    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9778494-18c5-4eb6-926b-92f69cbde312-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-b6ph5\" (UID: \"f9778494-18c5-4eb6-926b-92f69cbde312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5"
Oct 30 14:02:03.579302 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.578898    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9778494-18c5-4eb6-926b-92f69cbde312-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-b6ph5\" (UID: \"f9778494-18c5-4eb6-926b-92f69cbde312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5"
Oct 30 14:02:03.579302 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.578923    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9778494-18c5-4eb6-926b-92f69cbde312-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-b6ph5\" (UID: \"f9778494-18c5-4eb6-926b-92f69cbde312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5"
Oct 30 14:02:03.579302 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.578967    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9778494-18c5-4eb6-926b-92f69cbde312-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-b6ph5\" (UID: \"f9778494-18c5-4eb6-926b-92f69cbde312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5"
Oct 30 14:02:03.579302 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.578998    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2k8b\" (UID: \"762bb1a6-9339-4c49-bf97-1e160e7b0fc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b"
Oct 30 14:02:03.579302 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.579027    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2k8b\" (UID: \"762bb1a6-9339-4c49-bf97-1e160e7b0fc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b"
Oct 30 14:02:03.579302 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.579063    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2k8b\" (UID: \"762bb1a6-9339-4c49-bf97-1e160e7b0fc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b"
Oct 30 14:02:03.579302 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.579098    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9778494-18c5-4eb6-926b-92f69cbde312-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-b6ph5\" (UID: \"f9778494-18c5-4eb6-926b-92f69cbde312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5"
Oct 30 14:02:03.579302 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.579144    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9778494-18c5-4eb6-926b-92f69cbde312-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-b6ph5\" (UID: \"f9778494-18c5-4eb6-926b-92f69cbde312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5"
Oct 30 14:02:03.579302 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.579228    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2k8b\" (UID: \"762bb1a6-9339-4c49-bf97-1e160e7b0fc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b"
Oct 30 14:02:03.579856 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.579830    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnpl7\" (UniqueName: \"kubernetes.io/projected/f9778494-18c5-4eb6-926b-92f69cbde312-kube-api-access-cnpl7\") pod \"maas-default-gateway-openshift-default-67bf986f99-b6ph5\" (UID: \"f9778494-18c5-4eb6-926b-92f69cbde312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5"
Oct 30 14:02:03.579936 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.579900    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sspzw\" (UniqueName: \"kubernetes.io/projected/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-kube-api-access-sspzw\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2k8b\" (UID: \"762bb1a6-9339-4c49-bf97-1e160e7b0fc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b"
Oct 30 14:02:03.580016 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.579990    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9778494-18c5-4eb6-926b-92f69cbde312-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-b6ph5\" (UID: \"f9778494-18c5-4eb6-926b-92f69cbde312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5"
Oct 30 14:02:03.580082 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.579998    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2k8b\" (UID: \"762bb1a6-9339-4c49-bf97-1e160e7b0fc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b"
Oct 30 14:02:03.580328 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.580304    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9778494-18c5-4eb6-926b-92f69cbde312-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-b6ph5\" (UID: \"f9778494-18c5-4eb6-926b-92f69cbde312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5"
Oct 30 14:02:03.580789 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.580761    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2k8b\" (UID: \"762bb1a6-9339-4c49-bf97-1e160e7b0fc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b"
Oct 30 14:02:03.580914 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.580868    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9778494-18c5-4eb6-926b-92f69cbde312-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-b6ph5\" (UID: \"f9778494-18c5-4eb6-926b-92f69cbde312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5"
Oct 30 14:02:03.582168 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.581328    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2k8b\" (UID: \"762bb1a6-9339-4c49-bf97-1e160e7b0fc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b"
Oct 30 14:02:03.582168 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.581888    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2k8b\" (UID: \"762bb1a6-9339-4c49-bf97-1e160e7b0fc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b"
Oct 30 14:02:03.582168 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.582013    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9778494-18c5-4eb6-926b-92f69cbde312-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-b6ph5\" (UID: \"f9778494-18c5-4eb6-926b-92f69cbde312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5"
Oct 30 14:02:03.582168 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.581997    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9778494-18c5-4eb6-926b-92f69cbde312-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-b6ph5\" (UID: \"f9778494-18c5-4eb6-926b-92f69cbde312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5"
Oct 30 14:02:03.582451 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.582414    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2k8b\" (UID: \"762bb1a6-9339-4c49-bf97-1e160e7b0fc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b"
Oct 30 14:02:03.582612 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.582580    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2k8b\" (UID: \"762bb1a6-9339-4c49-bf97-1e160e7b0fc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b"
Oct 30 14:02:03.582692 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.582676    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9778494-18c5-4eb6-926b-92f69cbde312-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-b6ph5\" (UID: \"f9778494-18c5-4eb6-926b-92f69cbde312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5"
Oct 30 14:02:03.589471 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.582939    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9778494-18c5-4eb6-926b-92f69cbde312-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-b6ph5\" (UID: \"f9778494-18c5-4eb6-926b-92f69cbde312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5"
Oct 30 14:02:03.589471 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.583246    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2k8b\" (UID: \"762bb1a6-9339-4c49-bf97-1e160e7b0fc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b"
Oct 30 14:02:03.600334 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.600309    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9778494-18c5-4eb6-926b-92f69cbde312-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-b6ph5\" (UID: \"f9778494-18c5-4eb6-926b-92f69cbde312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5"
Oct 30 14:02:03.601203 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.601180    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnpl7\" (UniqueName: \"kubernetes.io/projected/f9778494-18c5-4eb6-926b-92f69cbde312-kube-api-access-cnpl7\") pod \"maas-default-gateway-openshift-default-67bf986f99-b6ph5\" (UID: \"f9778494-18c5-4eb6-926b-92f69cbde312\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5"
Oct 30 14:02:03.607119 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.607094    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2k8b\" (UID: \"762bb1a6-9339-4c49-bf97-1e160e7b0fc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b"
Oct 30 14:02:03.607419 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.607396    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sspzw\" (UniqueName: \"kubernetes.io/projected/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-kube-api-access-sspzw\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2k8b\" (UID: \"762bb1a6-9339-4c49-bf97-1e160e7b0fc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b"
Oct 30 14:02:03.730881 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.730836    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5"
Oct 30 14:02:03.779447 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:03.779415    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b"
Oct 30 14:02:04.067577 ip-10-0-134-31 kubenswrapper[5093]: W1030 14:02:04.067539    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9778494_18c5_4eb6_926b_92f69cbde312.slice/crio-6efb52b9c451bca3ac0545fca41c2f531d33ecf1e44e5d30af3349485f7cf690 WatchSource:0}: Error finding container 6efb52b9c451bca3ac0545fca41c2f531d33ecf1e44e5d30af3349485f7cf690: Status 404 returned error can't find the container with id 6efb52b9c451bca3ac0545fca41c2f531d33ecf1e44e5d30af3349485f7cf690
Oct 30 14:02:04.068218 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:04.068187    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5"]
Oct 30 14:02:04.071139 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:04.071085    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 14:02:04.071255 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:04.071194    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 14:02:04.071255 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:04.071241    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 14:02:04.116884 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:04.116857    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b"]
Oct 30 14:02:04.118039 ip-10-0-134-31 kubenswrapper[5093]: W1030 14:02:04.118004    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod762bb1a6_9339_4c49_bf97_1e160e7b0fc6.slice/crio-2c7c5fdaa090b87fe1938896a70a96d62e189a9035222507213e353240a276e0 WatchSource:0}: Error finding container 2c7c5fdaa090b87fe1938896a70a96d62e189a9035222507213e353240a276e0: Status 404 returned error can't find the container with id 2c7c5fdaa090b87fe1938896a70a96d62e189a9035222507213e353240a276e0
Oct 30 14:02:04.121462 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:04.121428    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 14:02:04.121544 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:04.121516    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 14:02:04.121591 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:04.121559    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 14:02:04.896309 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:04.896263    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5" event={"ID":"f9778494-18c5-4eb6-926b-92f69cbde312","Type":"ContainerStarted","Data":"36d337dfb735313c530f1c21e21c4e270679971d466c5794a3eb797d63bb2360"}
Oct 30 14:02:04.896529 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:04.896345    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5" event={"ID":"f9778494-18c5-4eb6-926b-92f69cbde312","Type":"ContainerStarted","Data":"6efb52b9c451bca3ac0545fca41c2f531d33ecf1e44e5d30af3349485f7cf690"}
Oct 30 14:02:04.898078 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:04.898046    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b" event={"ID":"762bb1a6-9339-4c49-bf97-1e160e7b0fc6","Type":"ContainerStarted","Data":"b3e545a30c051f6f4a59c8ea60a716fc6d246eac9915f0c924fcb1ac06cb46bb"}
Oct 30 14:02:04.898229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:04.898104    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b" event={"ID":"762bb1a6-9339-4c49-bf97-1e160e7b0fc6","Type":"ContainerStarted","Data":"2c7c5fdaa090b87fe1938896a70a96d62e189a9035222507213e353240a276e0"}
Oct 30 14:02:04.958008 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:04.957947    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5" podStartSLOduration=1.95792775 podStartE2EDuration="1.95792775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:02:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:02:04.952453566 +0000 UTC m=+3544.099720820" watchObservedRunningTime="2026-10-30 14:02:04.95792775 +0000 UTC m=+3544.105194995"
Oct 30 14:02:04.999405 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:04.999353    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b" podStartSLOduration=1.999337928 podStartE2EDuration="1.999337928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:02:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:02:04.996469047 +0000 UTC m=+3544.143736302" watchObservedRunningTime="2026-10-30 14:02:04.999337928 +0000 UTC m=+3544.146605168"
Oct 30 14:02:05.737153 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:05.733199    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5"
Oct 30 14:02:05.737153 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:05.735473    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-b6ph5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Oct 30 14:02:05.737153 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:05.735568    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5" podUID="f9778494-18c5-4eb6-926b-92f69cbde312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Oct 30 14:02:05.780060 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:05.780020    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b"
Oct 30 14:02:05.781760 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:05.781726    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-k2k8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Oct 30 14:02:05.781893 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:05.781778    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b" podUID="762bb1a6-9339-4c49-bf97-1e160e7b0fc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Oct 30 14:02:06.735922 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:06.735876    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-b6ph5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Oct 30 14:02:06.736268 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:06.736231    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5" podUID="f9778494-18c5-4eb6-926b-92f69cbde312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Oct 30 14:02:06.781590 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:06.781541    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-k2k8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Oct 30 14:02:06.782158 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:06.781612    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b" podUID="762bb1a6-9339-4c49-bf97-1e160e7b0fc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Oct 30 14:02:07.740153 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:07.736467    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-b6ph5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Oct 30 14:02:07.740153 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:07.736531    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5" podUID="f9778494-18c5-4eb6-926b-92f69cbde312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Oct 30 14:02:07.784155 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:07.783880    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-k2k8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Oct 30 14:02:07.784155 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:07.783945    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b" podUID="762bb1a6-9339-4c49-bf97-1e160e7b0fc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Oct 30 14:02:08.740626 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:08.740581    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-b6ph5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Oct 30 14:02:08.740949 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:08.740914    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5" podUID="f9778494-18c5-4eb6-926b-92f69cbde312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Oct 30 14:02:08.783156 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:08.782877    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-k2k8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Oct 30 14:02:08.783156 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:08.782943    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b" podUID="762bb1a6-9339-4c49-bf97-1e160e7b0fc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Oct 30 14:02:09.733156 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:09.731583    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-b6ph5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Oct 30 14:02:09.733156 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:09.731653    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5" podUID="f9778494-18c5-4eb6-926b-92f69cbde312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Oct 30 14:02:09.781154 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:09.780536    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-k2k8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Oct 30 14:02:09.781154 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:09.780602    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b" podUID="762bb1a6-9339-4c49-bf97-1e160e7b0fc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Oct 30 14:02:10.739693 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:10.739590    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-b6ph5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Oct 30 14:02:10.740307 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:10.740269    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5" podUID="f9778494-18c5-4eb6-926b-92f69cbde312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Oct 30 14:02:10.780060 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:10.780016    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-k2k8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Oct 30 14:02:10.780344 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:10.780317    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b" podUID="762bb1a6-9339-4c49-bf97-1e160e7b0fc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Oct 30 14:02:11.732330 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:11.732278    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-b6ph5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Oct 30 14:02:11.732557 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:11.732359    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5" podUID="f9778494-18c5-4eb6-926b-92f69cbde312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Oct 30 14:02:11.783488 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:11.782621    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-k2k8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Oct 30 14:02:11.783488 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:11.782685    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b" podUID="762bb1a6-9339-4c49-bf97-1e160e7b0fc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Oct 30 14:02:12.731892 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:12.731848    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-b6ph5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Oct 30 14:02:12.732088 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:12.731923    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5" podUID="f9778494-18c5-4eb6-926b-92f69cbde312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Oct 30 14:02:13.503430 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:13.503394    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b"
Oct 30 14:02:13.504908 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:13.504774    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b"
Oct 30 14:02:13.505841 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:13.505815    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b"
Oct 30 14:02:14.049634 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:14.049474    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5"
Oct 30 14:02:14.050953 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:14.050897    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5"
Oct 30 14:02:14.052000 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:14.051970    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5"
Oct 30 14:02:21.170006 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:02:21.169960    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 14:02:48.606659 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.606618    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2"]
Oct 30 14:02:48.613514 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.613488    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2"
Oct 30 14:02:48.650289 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.650243    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2"]
Oct 30 14:02:48.657085 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.657052    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2"]
Oct 30 14:02:48.662584 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.662551    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2"
Oct 30 14:02:48.675090 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.675055    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2"]
Oct 30 14:02:48.707502 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.705820    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2c83cc2-4c02-493a-948a-3a0ecf046a37-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwhx2\" (UID: \"d2c83cc2-4c02-493a-948a-3a0ecf046a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2"
Oct 30 14:02:48.707502 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.705972    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2c83cc2-4c02-493a-948a-3a0ecf046a37-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwhx2\" (UID: \"d2c83cc2-4c02-493a-948a-3a0ecf046a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2"
Oct 30 14:02:48.707502 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.706053    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2c83cc2-4c02-493a-948a-3a0ecf046a37-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwhx2\" (UID: \"d2c83cc2-4c02-493a-948a-3a0ecf046a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2"
Oct 30 14:02:48.707502 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.706103    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/926c412f-623c-489d-8100-c66c75a5f478-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-5lnv2\" (UID: \"926c412f-623c-489d-8100-c66c75a5f478\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2"
Oct 30 14:02:48.707502 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.706311    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c83cc2-4c02-493a-948a-3a0ecf046a37-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwhx2\" (UID: \"d2c83cc2-4c02-493a-948a-3a0ecf046a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2"
Oct 30 14:02:48.707502 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.706401    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djmrs\" (UniqueName: \"kubernetes.io/projected/926c412f-623c-489d-8100-c66c75a5f478-kube-api-access-djmrs\") pod \"maas-default-gateway-openshift-default-67bf986f99-5lnv2\" (UID: \"926c412f-623c-489d-8100-c66c75a5f478\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2"
Oct 30 14:02:48.707502 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.706465    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/926c412f-623c-489d-8100-c66c75a5f478-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-5lnv2\" (UID: \"926c412f-623c-489d-8100-c66c75a5f478\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2"
Oct 30 14:02:48.707502 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.706522    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/926c412f-623c-489d-8100-c66c75a5f478-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-5lnv2\" (UID: \"926c412f-623c-489d-8100-c66c75a5f478\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2"
Oct 30 14:02:48.707502 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.706588    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/926c412f-623c-489d-8100-c66c75a5f478-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-5lnv2\" (UID: \"926c412f-623c-489d-8100-c66c75a5f478\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2"
Oct 30 14:02:48.707502 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.706641    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvbpc\" (UniqueName: \"kubernetes.io/projected/d2c83cc2-4c02-493a-948a-3a0ecf046a37-kube-api-access-rvbpc\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwhx2\" (UID: \"d2c83cc2-4c02-493a-948a-3a0ecf046a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2"
Oct 30 14:02:48.707502 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.706690    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2c83cc2-4c02-493a-948a-3a0ecf046a37-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwhx2\" (UID: \"d2c83cc2-4c02-493a-948a-3a0ecf046a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2"
Oct 30 14:02:48.707502 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.706764    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/926c412f-623c-489d-8100-c66c75a5f478-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-5lnv2\" (UID: \"926c412f-623c-489d-8100-c66c75a5f478\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2"
Oct 30 14:02:48.707502 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.706901    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/926c412f-623c-489d-8100-c66c75a5f478-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-5lnv2\" (UID: \"926c412f-623c-489d-8100-c66c75a5f478\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2"
Oct 30 14:02:48.707502 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.706990    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c83cc2-4c02-493a-948a-3a0ecf046a37-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwhx2\" (UID: \"d2c83cc2-4c02-493a-948a-3a0ecf046a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2"
Oct 30 14:02:48.707502 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.707073    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2c83cc2-4c02-493a-948a-3a0ecf046a37-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwhx2\" (UID: \"d2c83cc2-4c02-493a-948a-3a0ecf046a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2"
Oct 30 14:02:48.707502 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.707123    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2c83cc2-4c02-493a-948a-3a0ecf046a37-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwhx2\" (UID: \"d2c83cc2-4c02-493a-948a-3a0ecf046a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2"
Oct 30 14:02:48.707502 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.707176    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/926c412f-623c-489d-8100-c66c75a5f478-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-5lnv2\" (UID: \"926c412f-623c-489d-8100-c66c75a5f478\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2"
Oct 30 14:02:48.707502 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.707252    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/926c412f-623c-489d-8100-c66c75a5f478-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-5lnv2\" (UID: \"926c412f-623c-489d-8100-c66c75a5f478\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2"
Oct 30 14:02:48.809266 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.809224    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/926c412f-623c-489d-8100-c66c75a5f478-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-5lnv2\" (UID: \"926c412f-623c-489d-8100-c66c75a5f478\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2"
Oct 30 14:02:48.809486 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.809291    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c83cc2-4c02-493a-948a-3a0ecf046a37-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwhx2\" (UID: \"d2c83cc2-4c02-493a-948a-3a0ecf046a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2"
Oct 30 14:02:48.809486 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.809338    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2c83cc2-4c02-493a-948a-3a0ecf046a37-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwhx2\" (UID: \"d2c83cc2-4c02-493a-948a-3a0ecf046a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2"
Oct 30 14:02:48.809486 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.809357    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2c83cc2-4c02-493a-948a-3a0ecf046a37-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwhx2\" (UID: \"d2c83cc2-4c02-493a-948a-3a0ecf046a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2"
Oct 30 14:02:48.809486 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.809377    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/926c412f-623c-489d-8100-c66c75a5f478-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-5lnv2\" (UID: \"926c412f-623c-489d-8100-c66c75a5f478\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2"
Oct 30 14:02:48.809486 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.809418    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/926c412f-623c-489d-8100-c66c75a5f478-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-5lnv2\" (UID: \"926c412f-623c-489d-8100-c66c75a5f478\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2"
Oct 30 14:02:48.809486 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.809447    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2c83cc2-4c02-493a-948a-3a0ecf046a37-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwhx2\" (UID: \"d2c83cc2-4c02-493a-948a-3a0ecf046a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2"
Oct 30 14:02:48.809835 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.809511    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2c83cc2-4c02-493a-948a-3a0ecf046a37-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwhx2\" (UID: \"d2c83cc2-4c02-493a-948a-3a0ecf046a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2"
Oct 30 14:02:48.809835 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.809541    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2c83cc2-4c02-493a-948a-3a0ecf046a37-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwhx2\" (UID: \"d2c83cc2-4c02-493a-948a-3a0ecf046a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2"
Oct 30 14:02:48.809835 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.809578    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/926c412f-623c-489d-8100-c66c75a5f478-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-5lnv2\" (UID: \"926c412f-623c-489d-8100-c66c75a5f478\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2"
Oct 30 14:02:48.809835 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.809607    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c83cc2-4c02-493a-948a-3a0ecf046a37-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwhx2\" (UID: \"d2c83cc2-4c02-493a-948a-3a0ecf046a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2"
Oct 30 14:02:48.809835 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.809644    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djmrs\" (UniqueName: \"kubernetes.io/projected/926c412f-623c-489d-8100-c66c75a5f478-kube-api-access-djmrs\") pod \"maas-default-gateway-openshift-default-67bf986f99-5lnv2\" (UID: \"926c412f-623c-489d-8100-c66c75a5f478\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2"
Oct 30 14:02:48.809835 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.809679    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/926c412f-623c-489d-8100-c66c75a5f478-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-5lnv2\" (UID: \"926c412f-623c-489d-8100-c66c75a5f478\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2"
Oct 30 14:02:48.809835 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.809699    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/926c412f-623c-489d-8100-c66c75a5f478-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-5lnv2\" (UID: \"926c412f-623c-489d-8100-c66c75a5f478\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2"
Oct 30 14:02:48.809835 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.809703    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2c83cc2-4c02-493a-948a-3a0ecf046a37-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwhx2\" (UID: \"d2c83cc2-4c02-493a-948a-3a0ecf046a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2"
Oct 30 14:02:48.809835 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.809714    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/926c412f-623c-489d-8100-c66c75a5f478-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-5lnv2\" (UID: \"926c412f-623c-489d-8100-c66c75a5f478\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2"
Oct 30 14:02:48.809835 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.809760    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c83cc2-4c02-493a-948a-3a0ecf046a37-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwhx2\" (UID: \"d2c83cc2-4c02-493a-948a-3a0ecf046a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2"
Oct 30 14:02:48.809835 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.809810    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/926c412f-623c-489d-8100-c66c75a5f478-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-5lnv2\" (UID: \"926c412f-623c-489d-8100-c66c75a5f478\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2"
Oct 30 14:02:48.810446 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.809856    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvbpc\" (UniqueName: \"kubernetes.io/projected/d2c83cc2-4c02-493a-948a-3a0ecf046a37-kube-api-access-rvbpc\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwhx2\" (UID: \"d2c83cc2-4c02-493a-948a-3a0ecf046a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2"
Oct 30 14:02:48.810446 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.809883    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2c83cc2-4c02-493a-948a-3a0ecf046a37-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwhx2\" (UID: \"d2c83cc2-4c02-493a-948a-3a0ecf046a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2"
Oct 30 14:02:48.810446 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.809934    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/926c412f-623c-489d-8100-c66c75a5f478-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-5lnv2\" (UID: \"926c412f-623c-489d-8100-c66c75a5f478\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2"
Oct 30 14:02:48.810446 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.809959    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2c83cc2-4c02-493a-948a-3a0ecf046a37-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwhx2\" (UID: \"d2c83cc2-4c02-493a-948a-3a0ecf046a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2"
Oct 30 14:02:48.810446 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.810013    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c83cc2-4c02-493a-948a-3a0ecf046a37-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwhx2\" (UID: \"d2c83cc2-4c02-493a-948a-3a0ecf046a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2"
Oct 30 14:02:48.810446 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.810072    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/926c412f-623c-489d-8100-c66c75a5f478-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-5lnv2\" (UID: \"926c412f-623c-489d-8100-c66c75a5f478\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2"
Oct 30 14:02:48.810446 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.810272    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/926c412f-623c-489d-8100-c66c75a5f478-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-5lnv2\" (UID: \"926c412f-623c-489d-8100-c66c75a5f478\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2"
Oct 30 14:02:48.810842 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.810480    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2c83cc2-4c02-493a-948a-3a0ecf046a37-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwhx2\" (UID: \"d2c83cc2-4c02-493a-948a-3a0ecf046a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2"
Oct 30 14:02:48.810842 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.810709    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/926c412f-623c-489d-8100-c66c75a5f478-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-5lnv2\" (UID: \"926c412f-623c-489d-8100-c66c75a5f478\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2"
Oct 30 14:02:48.810957 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.810910    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/926c412f-623c-489d-8100-c66c75a5f478-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-5lnv2\" (UID: \"926c412f-623c-489d-8100-c66c75a5f478\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2"
Oct 30 14:02:48.812059 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.812029    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2c83cc2-4c02-493a-948a-3a0ecf046a37-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwhx2\" (UID: \"d2c83cc2-4c02-493a-948a-3a0ecf046a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2"
Oct 30 14:02:48.812386 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.812366    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/926c412f-623c-489d-8100-c66c75a5f478-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-5lnv2\" (UID: \"926c412f-623c-489d-8100-c66c75a5f478\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2"
Oct 30 14:02:48.812674 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.812652    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2c83cc2-4c02-493a-948a-3a0ecf046a37-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwhx2\" (UID: \"d2c83cc2-4c02-493a-948a-3a0ecf046a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2"
Oct 30 14:02:48.812749 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.812736    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/926c412f-623c-489d-8100-c66c75a5f478-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-5lnv2\" (UID: \"926c412f-623c-489d-8100-c66c75a5f478\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2"
Oct 30 14:02:48.832963 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.832920    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/926c412f-623c-489d-8100-c66c75a5f478-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-5lnv2\" (UID: \"926c412f-623c-489d-8100-c66c75a5f478\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2"
Oct 30 14:02:48.833453 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.833428    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvbpc\" (UniqueName: \"kubernetes.io/projected/d2c83cc2-4c02-493a-948a-3a0ecf046a37-kube-api-access-rvbpc\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwhx2\" (UID: \"d2c83cc2-4c02-493a-948a-3a0ecf046a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2"
Oct 30 14:02:48.834730 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.834705    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djmrs\" (UniqueName: \"kubernetes.io/projected/926c412f-623c-489d-8100-c66c75a5f478-kube-api-access-djmrs\") pod \"maas-default-gateway-openshift-default-67bf986f99-5lnv2\" (UID: \"926c412f-623c-489d-8100-c66c75a5f478\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2"
Oct 30 14:02:48.835015 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.834991    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2c83cc2-4c02-493a-948a-3a0ecf046a37-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwhx2\" (UID: \"d2c83cc2-4c02-493a-948a-3a0ecf046a37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2"
Oct 30 14:02:48.931701 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:48.931661    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2"
Oct 30 14:02:49.009620 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:49.009582    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2"
Oct 30 14:02:49.289462 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:49.289427    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2"]
Oct 30 14:02:49.289836 ip-10-0-134-31 kubenswrapper[5093]: W1030 14:02:49.289813    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2c83cc2_4c02_493a_948a_3a0ecf046a37.slice/crio-0c941e74875452cea73df0b6d815603f6defac529c32b13a804caba000354e34 WatchSource:0}: Error finding container 0c941e74875452cea73df0b6d815603f6defac529c32b13a804caba000354e34: Status 404 returned error can't find the container with id 0c941e74875452cea73df0b6d815603f6defac529c32b13a804caba000354e34
Oct 30 14:02:49.293040 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:49.293005    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 14:02:49.293298 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:49.293077    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 14:02:49.293298 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:49.293107    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 14:02:49.325219 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:49.325182    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2" event={"ID":"d2c83cc2-4c02-493a-948a-3a0ecf046a37","Type":"ContainerStarted","Data":"0c941e74875452cea73df0b6d815603f6defac529c32b13a804caba000354e34"}
Oct 30 14:02:49.385486 ip-10-0-134-31 kubenswrapper[5093]: W1030 14:02:49.385436    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod926c412f_623c_489d_8100_c66c75a5f478.slice/crio-af782d5b6f1423bf1ad7f01e532745561de650a8708900ec5b4fc85ed841e8d9 WatchSource:0}: Error finding container af782d5b6f1423bf1ad7f01e532745561de650a8708900ec5b4fc85ed841e8d9: Status 404 returned error can't find the container with id af782d5b6f1423bf1ad7f01e532745561de650a8708900ec5b4fc85ed841e8d9
Oct 30 14:02:49.389219 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:49.385550    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2"]
Oct 30 14:02:49.389970 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:49.389932    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 14:02:49.390039 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:49.390016    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 14:02:49.390078 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:49.390067    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 14:02:50.394372 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:50.390833    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2" event={"ID":"d2c83cc2-4c02-493a-948a-3a0ecf046a37","Type":"ContainerStarted","Data":"8234e542f8c3186d92daa8ef0662dbea057aa22f4553b94d48752bdc7dddd7cb"}
Oct 30 14:02:50.397185 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:50.396421    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2" event={"ID":"926c412f-623c-489d-8100-c66c75a5f478","Type":"ContainerStarted","Data":"1b59835711224dbaf4410d797ab735072aa6903fe877d6734184f207c8ed1bc8"}
Oct 30 14:02:50.397185 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:50.396491    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2" event={"ID":"926c412f-623c-489d-8100-c66c75a5f478","Type":"ContainerStarted","Data":"af782d5b6f1423bf1ad7f01e532745561de650a8708900ec5b4fc85ed841e8d9"}
Oct 30 14:02:50.497206 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:50.497089    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2" podStartSLOduration=2.497068581 podStartE2EDuration="2.497068581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:02:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:02:50.467857821 +0000 UTC m=+3589.615125061" watchObservedRunningTime="2026-10-30 14:02:50.497068581 +0000 UTC m=+3589.644335825"
Oct 30 14:02:50.545426 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:50.525784    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2" podStartSLOduration=2.525764192 podStartE2EDuration="2.525764192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:02:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:02:50.520488315 +0000 UTC m=+3589.667755556" watchObservedRunningTime="2026-10-30 14:02:50.525764192 +0000 UTC m=+3589.673031435"
Oct 30 14:02:50.933156 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:50.932556    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2"
Oct 30 14:02:50.956157 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:50.939266    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fwhx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Oct 30 14:02:50.956157 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:50.939335    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2" podUID="d2c83cc2-4c02-493a-948a-3a0ecf046a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Oct 30 14:02:51.011699 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:51.011625    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2"
Oct 30 14:02:51.013106 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:51.013081    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-5lnv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Oct 30 14:02:51.013304 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:51.013284    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2" podUID="926c412f-623c-489d-8100-c66c75a5f478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Oct 30 14:02:51.202867 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:02:51.202772    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 14:02:51.932674 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:51.932571    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fwhx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Oct 30 14:02:51.935099 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:51.935059    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2" podUID="d2c83cc2-4c02-493a-948a-3a0ecf046a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Oct 30 14:02:52.012253 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:52.012218    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-5lnv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Oct 30 14:02:52.012445 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:52.012275    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2" podUID="926c412f-623c-489d-8100-c66c75a5f478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Oct 30 14:02:52.935285 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:52.935242    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fwhx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Oct 30 14:02:52.935884 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:52.935846    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2" podUID="d2c83cc2-4c02-493a-948a-3a0ecf046a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Oct 30 14:02:53.012544 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:53.012501    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-5lnv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Oct 30 14:02:53.012830 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:53.012555    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2" podUID="926c412f-623c-489d-8100-c66c75a5f478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Oct 30 14:02:53.946616 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:53.946572    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fwhx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Oct 30 14:02:53.947245 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:53.947207    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2" podUID="d2c83cc2-4c02-493a-948a-3a0ecf046a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Oct 30 14:02:54.012336 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:54.012302    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-5lnv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Oct 30 14:02:54.012625 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:54.012575    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2" podUID="926c412f-623c-489d-8100-c66c75a5f478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Oct 30 14:02:54.933572 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:54.933527    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fwhx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Oct 30 14:02:54.933908 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:54.933877    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2" podUID="d2c83cc2-4c02-493a-948a-3a0ecf046a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Oct 30 14:02:55.012389 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:55.012349    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-5lnv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Oct 30 14:02:55.012815 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:55.012425    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2" podUID="926c412f-623c-489d-8100-c66c75a5f478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Oct 30 14:02:55.932371 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:55.932320    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fwhx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Oct 30 14:02:55.932587 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:55.932400    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2" podUID="d2c83cc2-4c02-493a-948a-3a0ecf046a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Oct 30 14:02:56.012361 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:56.012317    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-5lnv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Oct 30 14:02:56.012551 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:56.012392    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2" podUID="926c412f-623c-489d-8100-c66c75a5f478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Oct 30 14:02:56.932680 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:56.932630    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fwhx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Oct 30 14:02:56.932887 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:56.932704    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2" podUID="d2c83cc2-4c02-493a-948a-3a0ecf046a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Oct 30 14:02:57.011981 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:57.011939    5093 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-5lnv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Oct 30 14:02:57.012182 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:57.012007    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2" podUID="926c412f-623c-489d-8100-c66c75a5f478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Oct 30 14:02:58.177508 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:58.177470    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2"
Oct 30 14:02:58.178292 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:58.178266    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2"
Oct 30 14:02:58.179400 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:58.179374    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2"
Oct 30 14:02:58.469874 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:58.469776    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2"
Oct 30 14:02:58.471237 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:58.471212    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2"
Oct 30 14:02:58.471878 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:02:58.471861    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2"
Oct 30 14:03:01.814375 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:01.814344    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jdj4_f93ee017-ab79-447c-90e0-5b85b71dba24/console-operator/1.log"
Oct 30 14:03:01.820483 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:01.820457    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tnm_3a675ac7-fbe1-449b-accf-bb0456ed0bd3/ovn-acl-logging/0.log"
Oct 30 14:03:01.824902 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:01.824875    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jdj4_f93ee017-ab79-447c-90e0-5b85b71dba24/console-operator/1.log"
Oct 30 14:03:01.831523 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:01.831501    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tnm_3a675ac7-fbe1-449b-accf-bb0456ed0bd3/ovn-acl-logging/0.log"
Oct 30 14:03:12.337309 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:12.336755    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ca0243/test-model-ca0243-kserve-65cdfc7597-8b765"]
Oct 30 14:03:15.520812 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:15.520750    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25" podUID="7805f89f-89d4-4754-b606-c3070f4df215" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:03:15.600204 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:15.598696    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg" podUID="ea079ac1-f502-4b5f-b16a-c91e2de98de8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:03:15.612340 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:15.612283    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw" podUID="ad37954e-1350-4f75-a63f-a0c0f29ae416" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:03:15.964156 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:15.959627    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx" podUID="ba156eeb-9955-4f9a-857b-2c4d502a5fcc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:03:21.168799 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:03:21.168751    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 14:03:26.653028 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:26.652988    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9b32c32a-6d586d78d7-qvxvx"]
Oct 30 14:03:26.667387 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:26.667346    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9b32c32a-6d586d78d7-qvxvx"
Oct 30 14:03:26.689485 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:26.689445    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-9b32c32a\""
Oct 30 14:03:26.693510 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:26.693465    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcxl5\" (UniqueName: \"kubernetes.io/projected/f4e930a4-18f6-49b6-938d-b2d4243643c7-kube-api-access-fcxl5\") pod \"payload-pre-processing-e2e-mt-9b32c32a-6d586d78d7-qvxvx\" (UID: \"f4e930a4-18f6-49b6-938d-b2d4243643c7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9b32c32a-6d586d78d7-qvxvx"
Oct 30 14:03:26.693671 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:26.693658    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4e930a4-18f6-49b6-938d-b2d4243643c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9b32c32a-6d586d78d7-qvxvx\" (UID: \"f4e930a4-18f6-49b6-938d-b2d4243643c7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9b32c32a-6d586d78d7-qvxvx"
Oct 30 14:03:26.741943 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:26.741336    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9b32c32a-6d586d78d7-qvxvx"]
Oct 30 14:03:26.749727 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:26.745975    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9b32c32a-75dd97f79d-t6tgs"]
Oct 30 14:03:26.773527 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:26.773494    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9b32c32a-75dd97f79d-t6tgs"
Oct 30 14:03:26.788887 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:26.787593    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9b32c32a-75dd97f79d-t6tgs"]
Oct 30 14:03:26.792065 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:26.792036    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-9b32c32a-dockercfg-zqkbr\""
Oct 30 14:03:26.794887 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:26.794847    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c822e501-9472-4c26-9be9-c308c69fccbb-plugins-config-volume\") pod \"payload-processing-e2e-mt-9b32c32a-75dd97f79d-t6tgs\" (UID: \"c822e501-9472-4c26-9be9-c308c69fccbb\") " pod="openshift-ingress/payload-processing-e2e-mt-9b32c32a-75dd97f79d-t6tgs"
Oct 30 14:03:26.795024 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:26.794909    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4e930a4-18f6-49b6-938d-b2d4243643c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9b32c32a-6d586d78d7-qvxvx\" (UID: \"f4e930a4-18f6-49b6-938d-b2d4243643c7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9b32c32a-6d586d78d7-qvxvx"
Oct 30 14:03:26.795024 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:26.794988    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcxl5\" (UniqueName: \"kubernetes.io/projected/f4e930a4-18f6-49b6-938d-b2d4243643c7-kube-api-access-fcxl5\") pod \"payload-pre-processing-e2e-mt-9b32c32a-6d586d78d7-qvxvx\" (UID: \"f4e930a4-18f6-49b6-938d-b2d4243643c7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9b32c32a-6d586d78d7-qvxvx"
Oct 30 14:03:26.795201 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:26.795075    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6qg6\" (UniqueName: \"kubernetes.io/projected/c822e501-9472-4c26-9be9-c308c69fccbb-kube-api-access-s6qg6\") pod \"payload-processing-e2e-mt-9b32c32a-75dd97f79d-t6tgs\" (UID: \"c822e501-9472-4c26-9be9-c308c69fccbb\") " pod="openshift-ingress/payload-processing-e2e-mt-9b32c32a-75dd97f79d-t6tgs"
Oct 30 14:03:26.796109 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:26.796083    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4e930a4-18f6-49b6-938d-b2d4243643c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9b32c32a-6d586d78d7-qvxvx\" (UID: \"f4e930a4-18f6-49b6-938d-b2d4243643c7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9b32c32a-6d586d78d7-qvxvx"
Oct 30 14:03:26.883404 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:26.883248    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcxl5\" (UniqueName: \"kubernetes.io/projected/f4e930a4-18f6-49b6-938d-b2d4243643c7-kube-api-access-fcxl5\") pod \"payload-pre-processing-e2e-mt-9b32c32a-6d586d78d7-qvxvx\" (UID: \"f4e930a4-18f6-49b6-938d-b2d4243643c7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9b32c32a-6d586d78d7-qvxvx"
Oct 30 14:03:26.897785 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:26.897732    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c822e501-9472-4c26-9be9-c308c69fccbb-plugins-config-volume\") pod \"payload-processing-e2e-mt-9b32c32a-75dd97f79d-t6tgs\" (UID: \"c822e501-9472-4c26-9be9-c308c69fccbb\") " pod="openshift-ingress/payload-processing-e2e-mt-9b32c32a-75dd97f79d-t6tgs"
Oct 30 14:03:26.899586 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:26.899556    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6qg6\" (UniqueName: \"kubernetes.io/projected/c822e501-9472-4c26-9be9-c308c69fccbb-kube-api-access-s6qg6\") pod \"payload-processing-e2e-mt-9b32c32a-75dd97f79d-t6tgs\" (UID: \"c822e501-9472-4c26-9be9-c308c69fccbb\") " pod="openshift-ingress/payload-processing-e2e-mt-9b32c32a-75dd97f79d-t6tgs"
Oct 30 14:03:26.901268 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:26.901242    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c822e501-9472-4c26-9be9-c308c69fccbb-plugins-config-volume\") pod \"payload-processing-e2e-mt-9b32c32a-75dd97f79d-t6tgs\" (UID: \"c822e501-9472-4c26-9be9-c308c69fccbb\") " pod="openshift-ingress/payload-processing-e2e-mt-9b32c32a-75dd97f79d-t6tgs"
Oct 30 14:03:26.929374 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:26.929216    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6qg6\" (UniqueName: \"kubernetes.io/projected/c822e501-9472-4c26-9be9-c308c69fccbb-kube-api-access-s6qg6\") pod \"payload-processing-e2e-mt-9b32c32a-75dd97f79d-t6tgs\" (UID: \"c822e501-9472-4c26-9be9-c308c69fccbb\") " pod="openshift-ingress/payload-processing-e2e-mt-9b32c32a-75dd97f79d-t6tgs"
Oct 30 14:03:26.995985 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:26.995948    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9b32c32a-6d586d78d7-qvxvx"
Oct 30 14:03:27.099679 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:27.099645    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9b32c32a-75dd97f79d-t6tgs"
Oct 30 14:03:27.333838 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:27.333808    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9b32c32a-6d586d78d7-qvxvx"]
Oct 30 14:03:27.334431 ip-10-0-134-31 kubenswrapper[5093]: W1030 14:03:27.334404    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4e930a4_18f6_49b6_938d_b2d4243643c7.slice/crio-7a9552092ddf4c77726d099a98b3f7c2e8f3eb5c5673e74dfc846795d2047058 WatchSource:0}: Error finding container 7a9552092ddf4c77726d099a98b3f7c2e8f3eb5c5673e74dfc846795d2047058: Status 404 returned error can't find the container with id 7a9552092ddf4c77726d099a98b3f7c2e8f3eb5c5673e74dfc846795d2047058
Oct 30 14:03:27.369236 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:27.369204    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-ca0243/test-model-ca0243-kserve-65cdfc7597-8b765" podUID="ec8e38bf-56e5-4699-9814-1d17423be437" containerName="main" containerID="cri-o://1070597f644c24611e6e3eaffb2e4e0617612aa7bffaadf584dfa38f9607a00a" gracePeriod=45
Oct 30 14:03:27.449281 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:27.449252    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9b32c32a-75dd97f79d-t6tgs"]
Oct 30 14:03:27.455954 ip-10-0-134-31 kubenswrapper[5093]: W1030 14:03:27.455922    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc822e501_9472_4c26_9be9_c308c69fccbb.slice/crio-f7fa68d4ad2342aa5266013b49a15a73cd320952c94c14227e2661dfc37976b3 WatchSource:0}: Error finding container f7fa68d4ad2342aa5266013b49a15a73cd320952c94c14227e2661dfc37976b3: Status 404 returned error can't find the container with id f7fa68d4ad2342aa5266013b49a15a73cd320952c94c14227e2661dfc37976b3
Oct 30 14:03:27.460536 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:27.460504    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-ca0243/test-model-ca0243-kserve-65cdfc7597-8b765" podUID="ec8e38bf-56e5-4699-9814-1d17423be437" containerName="main" probeResult="failure" output="Get \"https://10.132.0.189:8000/ready\": dial tcp 10.132.0.189:8000: connect: connection refused"
Oct 30 14:03:27.668756 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:27.668723    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ca0243/test-model-ca0243-kserve-65cdfc7597-8b765"
Oct 30 14:03:27.712635 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:27.712546    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec8e38bf-56e5-4699-9814-1d17423be437-kube-api-access-vtp9r\" (UniqueName: \"kubernetes.io/projected/ec8e38bf-56e5-4699-9814-1d17423be437-kube-api-access-vtp9r\") pod \"ec8e38bf-56e5-4699-9814-1d17423be437\" (UID: \"ec8e38bf-56e5-4699-9814-1d17423be437\") "
Oct 30 14:03:27.712635 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:27.712603    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec8e38bf-56e5-4699-9814-1d17423be437-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec8e38bf-56e5-4699-9814-1d17423be437-tls-certs\") pod \"ec8e38bf-56e5-4699-9814-1d17423be437\" (UID: \"ec8e38bf-56e5-4699-9814-1d17423be437\") "
Oct 30 14:03:27.712925 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:27.712894    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec8e38bf-56e5-4699-9814-1d17423be437-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec8e38bf-56e5-4699-9814-1d17423be437-tmp-dir\") pod \"ec8e38bf-56e5-4699-9814-1d17423be437\" (UID: \"ec8e38bf-56e5-4699-9814-1d17423be437\") "
Oct 30 14:03:27.713053 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:27.712934    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec8e38bf-56e5-4699-9814-1d17423be437-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec8e38bf-56e5-4699-9814-1d17423be437-dshm\") pod \"ec8e38bf-56e5-4699-9814-1d17423be437\" (UID: \"ec8e38bf-56e5-4699-9814-1d17423be437\") "
Oct 30 14:03:27.713053 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:27.712964    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec8e38bf-56e5-4699-9814-1d17423be437-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec8e38bf-56e5-4699-9814-1d17423be437-model-cache\") pod \"ec8e38bf-56e5-4699-9814-1d17423be437\" (UID: \"ec8e38bf-56e5-4699-9814-1d17423be437\") "
Oct 30 14:03:27.713053 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:27.713013    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec8e38bf-56e5-4699-9814-1d17423be437-home\" (UniqueName: \"kubernetes.io/empty-dir/ec8e38bf-56e5-4699-9814-1d17423be437-home\") pod \"ec8e38bf-56e5-4699-9814-1d17423be437\" (UID: \"ec8e38bf-56e5-4699-9814-1d17423be437\") "
Oct 30 14:03:27.713299 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:27.713260    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec8e38bf-56e5-4699-9814-1d17423be437-tmp-dir" pod "ec8e38bf-56e5-4699-9814-1d17423be437" (UID: "ec8e38bf-56e5-4699-9814-1d17423be437"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:03:27.713360 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:27.713301    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec8e38bf-56e5-4699-9814-1d17423be437-model-cache" pod "ec8e38bf-56e5-4699-9814-1d17423be437" (UID: "ec8e38bf-56e5-4699-9814-1d17423be437"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:03:27.713472 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:27.713453    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec8e38bf-56e5-4699-9814-1d17423be437-home" pod "ec8e38bf-56e5-4699-9814-1d17423be437" (UID: "ec8e38bf-56e5-4699-9814-1d17423be437"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:03:27.713790 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:27.713771    5093 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec8e38bf-56e5-4699-9814-1d17423be437-tmp-dir\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:03:27.713858 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:27.713794    5093 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec8e38bf-56e5-4699-9814-1d17423be437-model-cache\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:03:27.713858 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:27.713809    5093 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec8e38bf-56e5-4699-9814-1d17423be437-home\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:03:27.715157 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:27.715121    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec8e38bf-56e5-4699-9814-1d17423be437-kube-api-access-vtp9r" pod "ec8e38bf-56e5-4699-9814-1d17423be437" (UID: "ec8e38bf-56e5-4699-9814-1d17423be437"). InnerVolumeSpecName "kube-api-access-vtp9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:03:27.715304 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:27.715279    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec8e38bf-56e5-4699-9814-1d17423be437-dshm" pod "ec8e38bf-56e5-4699-9814-1d17423be437" (UID: "ec8e38bf-56e5-4699-9814-1d17423be437"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:03:27.715304 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:27.715287    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec8e38bf-56e5-4699-9814-1d17423be437-tls-certs" pod "ec8e38bf-56e5-4699-9814-1d17423be437" (UID: "ec8e38bf-56e5-4699-9814-1d17423be437"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:03:27.815206 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:27.815156    5093 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec8e38bf-56e5-4699-9814-1d17423be437-dshm\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:03:27.815206 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:27.815186    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtp9r\" (UniqueName: \"kubernetes.io/projected/ec8e38bf-56e5-4699-9814-1d17423be437-kube-api-access-vtp9r\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:03:27.815206 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:27.815197    5093 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec8e38bf-56e5-4699-9814-1d17423be437-tls-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:03:27.918279 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:27.918244    5093 generic.go:356] "Generic (PLEG): container finished" podID="ec8e38bf-56e5-4699-9814-1d17423be437" containerID="1070597f644c24611e6e3eaffb2e4e0617612aa7bffaadf584dfa38f9607a00a" exitCode=0
Oct 30 14:03:27.918473 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:27.918301    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ca0243/test-model-ca0243-kserve-65cdfc7597-8b765"
Oct 30 14:03:27.918473 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:27.918325    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ca0243/test-model-ca0243-kserve-65cdfc7597-8b765" event={"ID":"ec8e38bf-56e5-4699-9814-1d17423be437","Type":"ContainerDied","Data":"1070597f644c24611e6e3eaffb2e4e0617612aa7bffaadf584dfa38f9607a00a"}
Oct 30 14:03:27.918473 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:27.918365    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ca0243/test-model-ca0243-kserve-65cdfc7597-8b765" event={"ID":"ec8e38bf-56e5-4699-9814-1d17423be437","Type":"ContainerDied","Data":"9dd69aeda3983b972c7fb573ca91b229c605d67d47dc472aff9ea685aa75def8"}
Oct 30 14:03:27.918473 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:27.918390    5093 scope.go:122] "RemoveContainer" containerID="1070597f644c24611e6e3eaffb2e4e0617612aa7bffaadf584dfa38f9607a00a"
Oct 30 14:03:27.920659 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:27.920625    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9b32c32a-6d586d78d7-qvxvx" event={"ID":"f4e930a4-18f6-49b6-938d-b2d4243643c7","Type":"ContainerStarted","Data":"4e833ad80ec132d15d5b41fad3b633da5ea442a4c6bf1c951c6c0280ce4c48b9"}
Oct 30 14:03:27.920659 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:27.920686    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9b32c32a-6d586d78d7-qvxvx" event={"ID":"f4e930a4-18f6-49b6-938d-b2d4243643c7","Type":"ContainerStarted","Data":"7a9552092ddf4c77726d099a98b3f7c2e8f3eb5c5673e74dfc846795d2047058"}
Oct 30 14:03:27.920937 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:27.920717    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9b32c32a-6d586d78d7-qvxvx"
Oct 30 14:03:27.922965 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:27.922937    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9b32c32a-75dd97f79d-t6tgs" event={"ID":"c822e501-9472-4c26-9be9-c308c69fccbb","Type":"ContainerStarted","Data":"39a40002639e7e2c2148694586077e789d4b5cbda6308ac09e87341e28b93c5e"}
Oct 30 14:03:27.923124 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:27.922987    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9b32c32a-75dd97f79d-t6tgs" event={"ID":"c822e501-9472-4c26-9be9-c308c69fccbb","Type":"ContainerStarted","Data":"f7fa68d4ad2342aa5266013b49a15a73cd320952c94c14227e2661dfc37976b3"}
Oct 30 14:03:27.923124 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:27.923016    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-9b32c32a-75dd97f79d-t6tgs"
Oct 30 14:03:27.933218 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:27.933198    5093 scope.go:122] "RemoveContainer" containerID="1070597f644c24611e6e3eaffb2e4e0617612aa7bffaadf584dfa38f9607a00a"
Oct 30 14:03:27.933539 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:03:27.933518    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1070597f644c24611e6e3eaffb2e4e0617612aa7bffaadf584dfa38f9607a00a\": container with ID starting with 1070597f644c24611e6e3eaffb2e4e0617612aa7bffaadf584dfa38f9607a00a not found: ID does not exist" containerID="1070597f644c24611e6e3eaffb2e4e0617612aa7bffaadf584dfa38f9607a00a"
Oct 30 14:03:27.933601 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:27.933547    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1070597f644c24611e6e3eaffb2e4e0617612aa7bffaadf584dfa38f9607a00a"} err="failed to get container status \"1070597f644c24611e6e3eaffb2e4e0617612aa7bffaadf584dfa38f9607a00a\": rpc error: code = NotFound desc = could not find container \"1070597f644c24611e6e3eaffb2e4e0617612aa7bffaadf584dfa38f9607a00a\": container with ID starting with 1070597f644c24611e6e3eaffb2e4e0617612aa7bffaadf584dfa38f9607a00a not found: ID does not exist"
Oct 30 14:03:27.992728 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:27.992620    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ca0243/test-model-ca0243-kserve-65cdfc7597-8b765"]
Oct 30 14:03:28.006266 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:28.006228    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-ca0243/test-model-ca0243-kserve-65cdfc7597-8b765"]
Oct 30 14:03:28.047780 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:28.047716    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-9b32c32a-75dd97f79d-t6tgs" podStartSLOduration=2.047701788 podStartE2EDuration="2.047701788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:03:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:03:28.045808779 +0000 UTC m=+3627.193076020" watchObservedRunningTime="2026-10-30 14:03:28.047701788 +0000 UTC m=+3627.194969028"
Oct 30 14:03:28.102642 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:28.102581    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-9b32c32a-6d586d78d7-qvxvx" podStartSLOduration=2.102560814 podStartE2EDuration="2.102560814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:03:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:03:28.096863842 +0000 UTC m=+3627.244131082" watchObservedRunningTime="2026-10-30 14:03:28.102560814 +0000 UTC m=+3627.249828061"
Oct 30 14:03:29.002111 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:29.002065    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec8e38bf-56e5-4699-9814-1d17423be437" path="/var/lib/kubelet/pods/ec8e38bf-56e5-4699-9814-1d17423be437/volumes"
Oct 30 14:03:38.954378 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:38.954337    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9b32c32a-6d586d78d7-qvxvx"
Oct 30 14:03:38.976156 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:38.968326    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-9b32c32a-75dd97f79d-t6tgs"
Oct 30 14:03:39.523221 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:39.523170    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25" podUID="7805f89f-89d4-4754-b606-c3070f4df215" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:03:39.582015 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:39.563251    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb" podUID="c4b4e00e-f58d-4a4e-b7db-19f724360db1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:03:39.602304 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:39.602252    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw" podUID="ad37954e-1350-4f75-a63f-a0c0f29ae416" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:03:39.603623 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:39.602316    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg" podUID="ea079ac1-f502-4b5f-b16a-c91e2de98de8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:03:49.075949 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:49.075862    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-c684d2/sub-test-model-c684d2-kserve-575f6df8dc-z5xws"]
Oct 30 14:03:49.076670 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:49.076652    5093 cpu_manager.go:414] "containerMap: removing container" podUID="ec8e38bf-56e5-4699-9814-1d17423be437" containerName="main"
Oct 30 14:03:49.076727 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:49.076671    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec8e38bf-56e5-4699-9814-1d17423be437" containerName="main"
Oct 30 14:03:49.076812 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:49.076802    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec8e38bf-56e5-4699-9814-1d17423be437" containerName="main"
Oct 30 14:03:49.080695 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:49.080677    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-c684d2/sub-test-model-c684d2-kserve-575f6df8dc-z5xws"
Oct 30 14:03:49.099322 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:49.099284    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-c684d2\"/\"sub-test-model-c684d2-kserve-self-signed-certs\""
Oct 30 14:03:49.103156 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:49.103099    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-c684d2/sub-test-model-c684d2-kserve-575f6df8dc-z5xws"]
Oct 30 14:03:49.111329 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:49.111306    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-c684d2\"/\"default-dockercfg-nkjpp\""
Oct 30 14:03:49.111927 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:49.111905    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-c684d2\"/\"openshift-service-ca.crt\""
Oct 30 14:03:49.112109 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:49.112092    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-c684d2\"/\"kube-root-ca.crt\""
Oct 30 14:03:49.135117 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:49.135078    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34ea00b9-9cd9-4ee1-8704-fbd22eadaede-home\") pod \"sub-test-model-c684d2-kserve-575f6df8dc-z5xws\" (UID: \"34ea00b9-9cd9-4ee1-8704-fbd22eadaede\") " pod="ai-tenant-e2e-shared-a-w2-c684d2/sub-test-model-c684d2-kserve-575f6df8dc-z5xws"
Oct 30 14:03:49.135317 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:49.135196    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34ea00b9-9cd9-4ee1-8704-fbd22eadaede-dshm\") pod \"sub-test-model-c684d2-kserve-575f6df8dc-z5xws\" (UID: \"34ea00b9-9cd9-4ee1-8704-fbd22eadaede\") " pod="ai-tenant-e2e-shared-a-w2-c684d2/sub-test-model-c684d2-kserve-575f6df8dc-z5xws"
Oct 30 14:03:49.135317 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:49.135280    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34ea00b9-9cd9-4ee1-8704-fbd22eadaede-model-cache\") pod \"sub-test-model-c684d2-kserve-575f6df8dc-z5xws\" (UID: \"34ea00b9-9cd9-4ee1-8704-fbd22eadaede\") " pod="ai-tenant-e2e-shared-a-w2-c684d2/sub-test-model-c684d2-kserve-575f6df8dc-z5xws"
Oct 30 14:03:49.135461 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:49.135422    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmdbh\" (UniqueName: \"kubernetes.io/projected/34ea00b9-9cd9-4ee1-8704-fbd22eadaede-kube-api-access-jmdbh\") pod \"sub-test-model-c684d2-kserve-575f6df8dc-z5xws\" (UID: \"34ea00b9-9cd9-4ee1-8704-fbd22eadaede\") " pod="ai-tenant-e2e-shared-a-w2-c684d2/sub-test-model-c684d2-kserve-575f6df8dc-z5xws"
Oct 30 14:03:49.135461 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:49.135456    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34ea00b9-9cd9-4ee1-8704-fbd22eadaede-tls-certs\") pod \"sub-test-model-c684d2-kserve-575f6df8dc-z5xws\" (UID: \"34ea00b9-9cd9-4ee1-8704-fbd22eadaede\") " pod="ai-tenant-e2e-shared-a-w2-c684d2/sub-test-model-c684d2-kserve-575f6df8dc-z5xws"
Oct 30 14:03:49.135582 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:49.135530    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34ea00b9-9cd9-4ee1-8704-fbd22eadaede-tmp-dir\") pod \"sub-test-model-c684d2-kserve-575f6df8dc-z5xws\" (UID: \"34ea00b9-9cd9-4ee1-8704-fbd22eadaede\") " pod="ai-tenant-e2e-shared-a-w2-c684d2/sub-test-model-c684d2-kserve-575f6df8dc-z5xws"
Oct 30 14:03:49.236549 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:49.236514    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34ea00b9-9cd9-4ee1-8704-fbd22eadaede-home\") pod \"sub-test-model-c684d2-kserve-575f6df8dc-z5xws\" (UID: \"34ea00b9-9cd9-4ee1-8704-fbd22eadaede\") " pod="ai-tenant-e2e-shared-a-w2-c684d2/sub-test-model-c684d2-kserve-575f6df8dc-z5xws"
Oct 30 14:03:49.236748 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:49.236614    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34ea00b9-9cd9-4ee1-8704-fbd22eadaede-dshm\") pod \"sub-test-model-c684d2-kserve-575f6df8dc-z5xws\" (UID: \"34ea00b9-9cd9-4ee1-8704-fbd22eadaede\") " pod="ai-tenant-e2e-shared-a-w2-c684d2/sub-test-model-c684d2-kserve-575f6df8dc-z5xws"
Oct 30 14:03:49.236748 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:49.236638    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34ea00b9-9cd9-4ee1-8704-fbd22eadaede-model-cache\") pod \"sub-test-model-c684d2-kserve-575f6df8dc-z5xws\" (UID: \"34ea00b9-9cd9-4ee1-8704-fbd22eadaede\") " pod="ai-tenant-e2e-shared-a-w2-c684d2/sub-test-model-c684d2-kserve-575f6df8dc-z5xws"
Oct 30 14:03:49.236854 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:49.236770    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmdbh\" (UniqueName: \"kubernetes.io/projected/34ea00b9-9cd9-4ee1-8704-fbd22eadaede-kube-api-access-jmdbh\") pod \"sub-test-model-c684d2-kserve-575f6df8dc-z5xws\" (UID: \"34ea00b9-9cd9-4ee1-8704-fbd22eadaede\") " pod="ai-tenant-e2e-shared-a-w2-c684d2/sub-test-model-c684d2-kserve-575f6df8dc-z5xws"
Oct 30 14:03:49.236854 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:49.236814    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34ea00b9-9cd9-4ee1-8704-fbd22eadaede-tls-certs\") pod \"sub-test-model-c684d2-kserve-575f6df8dc-z5xws\" (UID: \"34ea00b9-9cd9-4ee1-8704-fbd22eadaede\") " pod="ai-tenant-e2e-shared-a-w2-c684d2/sub-test-model-c684d2-kserve-575f6df8dc-z5xws"
Oct 30 14:03:49.236940 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:49.236868    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34ea00b9-9cd9-4ee1-8704-fbd22eadaede-tmp-dir\") pod \"sub-test-model-c684d2-kserve-575f6df8dc-z5xws\" (UID: \"34ea00b9-9cd9-4ee1-8704-fbd22eadaede\") " pod="ai-tenant-e2e-shared-a-w2-c684d2/sub-test-model-c684d2-kserve-575f6df8dc-z5xws"
Oct 30 14:03:49.237031 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:49.237005    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34ea00b9-9cd9-4ee1-8704-fbd22eadaede-home\") pod \"sub-test-model-c684d2-kserve-575f6df8dc-z5xws\" (UID: \"34ea00b9-9cd9-4ee1-8704-fbd22eadaede\") " pod="ai-tenant-e2e-shared-a-w2-c684d2/sub-test-model-c684d2-kserve-575f6df8dc-z5xws"
Oct 30 14:03:49.237180 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:49.237110    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34ea00b9-9cd9-4ee1-8704-fbd22eadaede-model-cache\") pod \"sub-test-model-c684d2-kserve-575f6df8dc-z5xws\" (UID: \"34ea00b9-9cd9-4ee1-8704-fbd22eadaede\") " pod="ai-tenant-e2e-shared-a-w2-c684d2/sub-test-model-c684d2-kserve-575f6df8dc-z5xws"
Oct 30 14:03:49.237290 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:49.237269    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34ea00b9-9cd9-4ee1-8704-fbd22eadaede-tmp-dir\") pod \"sub-test-model-c684d2-kserve-575f6df8dc-z5xws\" (UID: \"34ea00b9-9cd9-4ee1-8704-fbd22eadaede\") " pod="ai-tenant-e2e-shared-a-w2-c684d2/sub-test-model-c684d2-kserve-575f6df8dc-z5xws"
Oct 30 14:03:49.239426 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:49.239403    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34ea00b9-9cd9-4ee1-8704-fbd22eadaede-dshm\") pod \"sub-test-model-c684d2-kserve-575f6df8dc-z5xws\" (UID: \"34ea00b9-9cd9-4ee1-8704-fbd22eadaede\") " pod="ai-tenant-e2e-shared-a-w2-c684d2/sub-test-model-c684d2-kserve-575f6df8dc-z5xws"
Oct 30 14:03:49.239550 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:49.239526    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34ea00b9-9cd9-4ee1-8704-fbd22eadaede-tls-certs\") pod \"sub-test-model-c684d2-kserve-575f6df8dc-z5xws\" (UID: \"34ea00b9-9cd9-4ee1-8704-fbd22eadaede\") " pod="ai-tenant-e2e-shared-a-w2-c684d2/sub-test-model-c684d2-kserve-575f6df8dc-z5xws"
Oct 30 14:03:49.263836 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:49.263793    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmdbh\" (UniqueName: \"kubernetes.io/projected/34ea00b9-9cd9-4ee1-8704-fbd22eadaede-kube-api-access-jmdbh\") pod \"sub-test-model-c684d2-kserve-575f6df8dc-z5xws\" (UID: \"34ea00b9-9cd9-4ee1-8704-fbd22eadaede\") " pod="ai-tenant-e2e-shared-a-w2-c684d2/sub-test-model-c684d2-kserve-575f6df8dc-z5xws"
Oct 30 14:03:49.411513 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:49.395300    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-c684d2/sub-test-model-c684d2-kserve-575f6df8dc-z5xws"
Oct 30 14:03:49.749750 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:49.749714    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-c684d2/sub-test-model-c684d2-kserve-575f6df8dc-z5xws"]
Oct 30 14:03:49.753972 ip-10-0-134-31 kubenswrapper[5093]: W1030 14:03:49.753934    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34ea00b9_9cd9_4ee1_8704_fbd22eadaede.slice/crio-f5063c16a823175a2030f6e2d51c073ad3694c8d674dce00a068ee8491d02810 WatchSource:0}: Error finding container f5063c16a823175a2030f6e2d51c073ad3694c8d674dce00a068ee8491d02810: Status 404 returned error can't find the container with id f5063c16a823175a2030f6e2d51c073ad3694c8d674dce00a068ee8491d02810
Oct 30 14:03:50.427879 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:50.427839    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-c684d2/sub-test-model-c684d2-kserve-575f6df8dc-z5xws" event={"ID":"34ea00b9-9cd9-4ee1-8704-fbd22eadaede","Type":"ContainerStarted","Data":"323afc65de4d0e2f37e669c14455096fc42b7868af00f7893a82336e96b54120"}
Oct 30 14:03:50.428307 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:50.427892    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-c684d2/sub-test-model-c684d2-kserve-575f6df8dc-z5xws" event={"ID":"34ea00b9-9cd9-4ee1-8704-fbd22eadaede","Type":"ContainerStarted","Data":"f5063c16a823175a2030f6e2d51c073ad3694c8d674dce00a068ee8491d02810"}
Oct 30 14:03:50.493270 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:50.493209    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-c684d2/sub-test-model-c684d2-kserve-575f6df8dc-z5xws" podStartSLOduration=1.493193631 podStartE2EDuration="1.493193631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:03:50.490047093 +0000 UTC m=+3649.637314332" watchObservedRunningTime="2026-10-30 14:03:50.493193631 +0000 UTC m=+3649.640460913"
Oct 30 14:03:51.168721 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:03:51.168681    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 14:03:51.400774 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:51.400732    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-c684d2/sub-test-model-c684d2-kserve-575f6df8dc-z5xws"
Oct 30 14:03:59.400522 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:59.400487    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-c684d2/sub-test-model-c684d2-kserve-575f6df8dc-z5xws"
Oct 30 14:03:59.413748 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:59.413701    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-c684d2/sub-test-model-c684d2-kserve-575f6df8dc-z5xws"
Oct 30 14:03:59.519935 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:03:59.519904    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-c684d2/sub-test-model-c684d2-kserve-575f6df8dc-z5xws"
Oct 30 14:04:07.265809 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:07.265767    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-310546-6db5dd49d6-xfhp4"]
Oct 30 14:04:07.334868 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:07.334829    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-310546-7c9f444fc9-xw4r4"]
Oct 30 14:04:08.641333 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:08.641292    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d724c8-54d88bfbbb-t6j9r"]
Oct 30 14:04:08.641839 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:08.641559    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d724c8-54d88bfbbb-t6j9r" podUID="b676aeef-6a09-4e04-929d-6388b8034eac" containerName="maas-api" containerID="cri-o://7f3264e45a87ad8670a358ea9cd775157ff1305b2bd76c0def97bc5fdb44905a" gracePeriod=30
Oct 30 14:04:09.018908 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:09.018878    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d724c8-54d88bfbbb-t6j9r"
Oct 30 14:04:09.145662 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:09.141834    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b676aeef-6a09-4e04-929d-6388b8034eac-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b676aeef-6a09-4e04-929d-6388b8034eac-maas-api-tls\") pod \"b676aeef-6a09-4e04-929d-6388b8034eac\" (UID: \"b676aeef-6a09-4e04-929d-6388b8034eac\") "
Oct 30 14:04:09.145662 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:09.141961    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b676aeef-6a09-4e04-929d-6388b8034eac-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b676aeef-6a09-4e04-929d-6388b8034eac-metrics-certs\") pod \"b676aeef-6a09-4e04-929d-6388b8034eac\" (UID: \"b676aeef-6a09-4e04-929d-6388b8034eac\") "
Oct 30 14:04:09.145662 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:09.142116    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b676aeef-6a09-4e04-929d-6388b8034eac-kube-api-access-qdvnj\" (UniqueName: \"kubernetes.io/projected/b676aeef-6a09-4e04-929d-6388b8034eac-kube-api-access-qdvnj\") pod \"b676aeef-6a09-4e04-929d-6388b8034eac\" (UID: \"b676aeef-6a09-4e04-929d-6388b8034eac\") "
Oct 30 14:04:09.145662 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:09.142234    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b676aeef-6a09-4e04-929d-6388b8034eac-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b676aeef-6a09-4e04-929d-6388b8034eac-openshift-service-ca\") pod \"b676aeef-6a09-4e04-929d-6388b8034eac\" (UID: \"b676aeef-6a09-4e04-929d-6388b8034eac\") "
Oct 30 14:04:09.145662 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:09.143274    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b676aeef-6a09-4e04-929d-6388b8034eac-openshift-service-ca" pod "b676aeef-6a09-4e04-929d-6388b8034eac" (UID: "b676aeef-6a09-4e04-929d-6388b8034eac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:04:09.147769 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:09.147742    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b676aeef-6a09-4e04-929d-6388b8034eac-metrics-certs" pod "b676aeef-6a09-4e04-929d-6388b8034eac" (UID: "b676aeef-6a09-4e04-929d-6388b8034eac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:04:09.147949 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:09.147786    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b676aeef-6a09-4e04-929d-6388b8034eac-maas-api-tls" pod "b676aeef-6a09-4e04-929d-6388b8034eac" (UID: "b676aeef-6a09-4e04-929d-6388b8034eac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:04:09.148624 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:09.148593    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b676aeef-6a09-4e04-929d-6388b8034eac-kube-api-access-qdvnj" pod "b676aeef-6a09-4e04-929d-6388b8034eac" (UID: "b676aeef-6a09-4e04-929d-6388b8034eac"). InnerVolumeSpecName "kube-api-access-qdvnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:04:09.244305 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:09.244181    5093 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b676aeef-6a09-4e04-929d-6388b8034eac-metrics-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:04:09.244305 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:09.244222    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdvnj\" (UniqueName: \"kubernetes.io/projected/b676aeef-6a09-4e04-929d-6388b8034eac-kube-api-access-qdvnj\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:04:09.244305 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:09.244238    5093 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b676aeef-6a09-4e04-929d-6388b8034eac-openshift-service-ca\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:04:09.244305 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:09.244252    5093 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b676aeef-6a09-4e04-929d-6388b8034eac-maas-api-tls\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:04:09.599934 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:09.599831    5093 generic.go:356] "Generic (PLEG): container finished" podID="b676aeef-6a09-4e04-929d-6388b8034eac" containerID="7f3264e45a87ad8670a358ea9cd775157ff1305b2bd76c0def97bc5fdb44905a" exitCode=0
Oct 30 14:04:09.599934 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:09.599881    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d724c8-54d88bfbbb-t6j9r"
Oct 30 14:04:09.599934 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:09.599906    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d724c8-54d88bfbbb-t6j9r" event={"ID":"b676aeef-6a09-4e04-929d-6388b8034eac","Type":"ContainerDied","Data":"7f3264e45a87ad8670a358ea9cd775157ff1305b2bd76c0def97bc5fdb44905a"}
Oct 30 14:04:09.600256 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:09.599965    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d724c8-54d88bfbbb-t6j9r" event={"ID":"b676aeef-6a09-4e04-929d-6388b8034eac","Type":"ContainerDied","Data":"f1f1c6fbf2018c0008b83e97d3bfa577cd5bf3c1a62cc99bbe6588ce149566e9"}
Oct 30 14:04:09.600256 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:09.599988    5093 scope.go:122] "RemoveContainer" containerID="7f3264e45a87ad8670a358ea9cd775157ff1305b2bd76c0def97bc5fdb44905a"
Oct 30 14:04:09.614710 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:09.614686    5093 scope.go:122] "RemoveContainer" containerID="7f3264e45a87ad8670a358ea9cd775157ff1305b2bd76c0def97bc5fdb44905a"
Oct 30 14:04:09.615065 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:04:09.615044    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f3264e45a87ad8670a358ea9cd775157ff1305b2bd76c0def97bc5fdb44905a\": container with ID starting with 7f3264e45a87ad8670a358ea9cd775157ff1305b2bd76c0def97bc5fdb44905a not found: ID does not exist" containerID="7f3264e45a87ad8670a358ea9cd775157ff1305b2bd76c0def97bc5fdb44905a"
Oct 30 14:04:09.615120 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:09.615074    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f3264e45a87ad8670a358ea9cd775157ff1305b2bd76c0def97bc5fdb44905a"} err="failed to get container status \"7f3264e45a87ad8670a358ea9cd775157ff1305b2bd76c0def97bc5fdb44905a\": rpc error: code = NotFound desc = could not find container \"7f3264e45a87ad8670a358ea9cd775157ff1305b2bd76c0def97bc5fdb44905a\": container with ID starting with 7f3264e45a87ad8670a358ea9cd775157ff1305b2bd76c0def97bc5fdb44905a not found: ID does not exist"
Oct 30 14:04:09.667208 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:09.667165    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d724c8-54d88bfbbb-t6j9r"]
Oct 30 14:04:09.679618 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:09.679581    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d724c8-54d88bfbbb-t6j9r"]
Oct 30 14:04:11.001746 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:11.001711    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b676aeef-6a09-4e04-929d-6388b8034eac" path="/var/lib/kubelet/pods/b676aeef-6a09-4e04-929d-6388b8034eac/volumes"
Oct 30 14:04:12.267190 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:12.267125    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-310546-6db5dd49d6-xfhp4" podUID="958c7031-f31a-4c34-9dd3-d01530719385" containerName="payload-processing" containerID="cri-o://8d81c0d9f185a07d983112478f4398e2ebc3ac4e9dc7dfb340f9423b802d1590" gracePeriod=25
Oct 30 14:04:12.335810 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:12.335764    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-310546-7c9f444fc9-xw4r4" podUID="63b21cf6-7b3b-48b6-88a1-d03e804d33fa" containerName="payload-pre-processing" containerID="cri-o://b4a90681e85564f7bb07f692e80acd481e000126e36818af5e23963dd4df4ff5" gracePeriod=25
Oct 30 14:04:20.958740 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:20.958691    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-310546-7c9f444fc9-xw4r4" podUID="63b21cf6-7b3b-48b6-88a1-d03e804d33fa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:04:21.169376 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:04:21.169331    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 14:04:21.975260 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:21.975216    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-310546-6db5dd49d6-xfhp4" podUID="958c7031-f31a-4c34-9dd3-d01530719385" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:04:30.958744 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:30.958698    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-310546-7c9f444fc9-xw4r4" podUID="63b21cf6-7b3b-48b6-88a1-d03e804d33fa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:04:31.975823 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:31.975782    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-310546-6db5dd49d6-xfhp4" podUID="958c7031-f31a-4c34-9dd3-d01530719385" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:04:32.714922 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:32.714458    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-310546-7c9f444fc9-xw4r4"
Oct 30 14:04:32.813907 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:32.813877    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-310546-6db5dd49d6-xfhp4"
Oct 30 14:04:32.826953 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:32.826926    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63b21cf6-7b3b-48b6-88a1-d03e804d33fa-kube-api-access-45ths\" (UniqueName: \"kubernetes.io/projected/63b21cf6-7b3b-48b6-88a1-d03e804d33fa-kube-api-access-45ths\") pod \"63b21cf6-7b3b-48b6-88a1-d03e804d33fa\" (UID: \"63b21cf6-7b3b-48b6-88a1-d03e804d33fa\") "
Oct 30 14:04:32.827164 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:32.827008    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63b21cf6-7b3b-48b6-88a1-d03e804d33fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63b21cf6-7b3b-48b6-88a1-d03e804d33fa-plugins-config-volume\") pod \"63b21cf6-7b3b-48b6-88a1-d03e804d33fa\" (UID: \"63b21cf6-7b3b-48b6-88a1-d03e804d33fa\") "
Oct 30 14:04:32.827527 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:32.827495    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63b21cf6-7b3b-48b6-88a1-d03e804d33fa-plugins-config-volume" pod "63b21cf6-7b3b-48b6-88a1-d03e804d33fa" (UID: "63b21cf6-7b3b-48b6-88a1-d03e804d33fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:04:32.827761 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:32.827735    5093 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63b21cf6-7b3b-48b6-88a1-d03e804d33fa-plugins-config-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:04:32.829299 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:32.829269    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63b21cf6-7b3b-48b6-88a1-d03e804d33fa-kube-api-access-45ths" pod "63b21cf6-7b3b-48b6-88a1-d03e804d33fa" (UID: "63b21cf6-7b3b-48b6-88a1-d03e804d33fa"). InnerVolumeSpecName "kube-api-access-45ths". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:04:32.858675 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:32.858627    5093 generic.go:356] "Generic (PLEG): container finished" podID="63b21cf6-7b3b-48b6-88a1-d03e804d33fa" containerID="b4a90681e85564f7bb07f692e80acd481e000126e36818af5e23963dd4df4ff5" exitCode=0
Oct 30 14:04:32.858888 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:32.858759    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-310546-7c9f444fc9-xw4r4" event={"ID":"63b21cf6-7b3b-48b6-88a1-d03e804d33fa","Type":"ContainerDied","Data":"b4a90681e85564f7bb07f692e80acd481e000126e36818af5e23963dd4df4ff5"}
Oct 30 14:04:32.858888 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:32.858815    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-310546-7c9f444fc9-xw4r4" event={"ID":"63b21cf6-7b3b-48b6-88a1-d03e804d33fa","Type":"ContainerDied","Data":"5a68d294b32223b74100fd5fd81b71e01924b88348abfcd9a644daa0e2098856"}
Oct 30 14:04:32.858888 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:32.858848    5093 scope.go:122] "RemoveContainer" containerID="b4a90681e85564f7bb07f692e80acd481e000126e36818af5e23963dd4df4ff5"
Oct 30 14:04:32.859082 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:32.859018    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-310546-7c9f444fc9-xw4r4"
Oct 30 14:04:32.866908 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:32.864982    5093 generic.go:356] "Generic (PLEG): container finished" podID="958c7031-f31a-4c34-9dd3-d01530719385" containerID="8d81c0d9f185a07d983112478f4398e2ebc3ac4e9dc7dfb340f9423b802d1590" exitCode=0
Oct 30 14:04:32.866908 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:32.865092    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-310546-6db5dd49d6-xfhp4"
Oct 30 14:04:32.866908 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:32.865270    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-310546-6db5dd49d6-xfhp4" event={"ID":"958c7031-f31a-4c34-9dd3-d01530719385","Type":"ContainerDied","Data":"8d81c0d9f185a07d983112478f4398e2ebc3ac4e9dc7dfb340f9423b802d1590"}
Oct 30 14:04:32.866908 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:32.865341    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-310546-6db5dd49d6-xfhp4" event={"ID":"958c7031-f31a-4c34-9dd3-d01530719385","Type":"ContainerDied","Data":"0e3f0c98a7acaea5933b92cd2d4860ad89b0bbfd6e59c4e9956204b43ce02c07"}
Oct 30 14:04:32.890402 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:32.890363    5093 scope.go:122] "RemoveContainer" containerID="b4a90681e85564f7bb07f692e80acd481e000126e36818af5e23963dd4df4ff5"
Oct 30 14:04:32.890825 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:04:32.890796    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4a90681e85564f7bb07f692e80acd481e000126e36818af5e23963dd4df4ff5\": container with ID starting with b4a90681e85564f7bb07f692e80acd481e000126e36818af5e23963dd4df4ff5 not found: ID does not exist" containerID="b4a90681e85564f7bb07f692e80acd481e000126e36818af5e23963dd4df4ff5"
Oct 30 14:04:32.890894 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:32.890839    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4a90681e85564f7bb07f692e80acd481e000126e36818af5e23963dd4df4ff5"} err="failed to get container status \"b4a90681e85564f7bb07f692e80acd481e000126e36818af5e23963dd4df4ff5\": rpc error: code = NotFound desc = could not find container \"b4a90681e85564f7bb07f692e80acd481e000126e36818af5e23963dd4df4ff5\": container with ID starting with b4a90681e85564f7bb07f692e80acd481e000126e36818af5e23963dd4df4ff5 not found: ID does not exist"
Oct 30 14:04:32.890894 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:32.890865    5093 scope.go:122] "RemoveContainer" containerID="8d81c0d9f185a07d983112478f4398e2ebc3ac4e9dc7dfb340f9423b802d1590"
Oct 30 14:04:32.902105 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:32.902077    5093 scope.go:122] "RemoveContainer" containerID="8d81c0d9f185a07d983112478f4398e2ebc3ac4e9dc7dfb340f9423b802d1590"
Oct 30 14:04:32.902448 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:04:32.902421    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d81c0d9f185a07d983112478f4398e2ebc3ac4e9dc7dfb340f9423b802d1590\": container with ID starting with 8d81c0d9f185a07d983112478f4398e2ebc3ac4e9dc7dfb340f9423b802d1590 not found: ID does not exist" containerID="8d81c0d9f185a07d983112478f4398e2ebc3ac4e9dc7dfb340f9423b802d1590"
Oct 30 14:04:32.902518 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:32.902460    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d81c0d9f185a07d983112478f4398e2ebc3ac4e9dc7dfb340f9423b802d1590"} err="failed to get container status \"8d81c0d9f185a07d983112478f4398e2ebc3ac4e9dc7dfb340f9423b802d1590\": rpc error: code = NotFound desc = could not find container \"8d81c0d9f185a07d983112478f4398e2ebc3ac4e9dc7dfb340f9423b802d1590\": container with ID starting with 8d81c0d9f185a07d983112478f4398e2ebc3ac4e9dc7dfb340f9423b802d1590 not found: ID does not exist"
Oct 30 14:04:32.929220 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:32.929185    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/958c7031-f31a-4c34-9dd3-d01530719385-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/958c7031-f31a-4c34-9dd3-d01530719385-plugins-config-volume\") pod \"958c7031-f31a-4c34-9dd3-d01530719385\" (UID: \"958c7031-f31a-4c34-9dd3-d01530719385\") "
Oct 30 14:04:32.929416 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:32.929388    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/958c7031-f31a-4c34-9dd3-d01530719385-kube-api-access-whvrq\" (UniqueName: \"kubernetes.io/projected/958c7031-f31a-4c34-9dd3-d01530719385-kube-api-access-whvrq\") pod \"958c7031-f31a-4c34-9dd3-d01530719385\" (UID: \"958c7031-f31a-4c34-9dd3-d01530719385\") "
Oct 30 14:04:32.929708 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:32.929681    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/958c7031-f31a-4c34-9dd3-d01530719385-plugins-config-volume" pod "958c7031-f31a-4c34-9dd3-d01530719385" (UID: "958c7031-f31a-4c34-9dd3-d01530719385"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:04:32.929977 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:32.929956    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45ths\" (UniqueName: \"kubernetes.io/projected/63b21cf6-7b3b-48b6-88a1-d03e804d33fa-kube-api-access-45ths\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:04:32.930039 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:32.929983    5093 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/958c7031-f31a-4c34-9dd3-d01530719385-plugins-config-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:04:32.931515 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:32.931493    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/958c7031-f31a-4c34-9dd3-d01530719385-kube-api-access-whvrq" pod "958c7031-f31a-4c34-9dd3-d01530719385" (UID: "958c7031-f31a-4c34-9dd3-d01530719385"). InnerVolumeSpecName "kube-api-access-whvrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:04:33.013410 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:33.013369    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-310546-7c9f444fc9-xw4r4"]
Oct 30 14:04:33.019818 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:33.019785    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-310546-7c9f444fc9-xw4r4"]
Oct 30 14:04:33.031351 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:33.031311    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whvrq\" (UniqueName: \"kubernetes.io/projected/958c7031-f31a-4c34-9dd3-d01530719385-kube-api-access-whvrq\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:04:33.223741 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:33.223690    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-310546-6db5dd49d6-xfhp4"]
Oct 30 14:04:33.227830 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:33.227796    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-310546-6db5dd49d6-xfhp4"]
Oct 30 14:04:34.781951 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:34.781908    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-310546-7f57cb9f95-5ghvp"]
Oct 30 14:04:34.782600 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:34.782223    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-310546-7f57cb9f95-5ghvp" podUID="b819faba-b9f4-488f-bfa0-4da40fb59313" containerName="maas-api" containerID="cri-o://58a1c5009b3015c501c0e183d602e2576f1bc53570e7616fbb66fcd1c8838838" gracePeriod=30
Oct 30 14:04:35.010211 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:35.010177    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63b21cf6-7b3b-48b6-88a1-d03e804d33fa" path="/var/lib/kubelet/pods/63b21cf6-7b3b-48b6-88a1-d03e804d33fa/volumes"
Oct 30 14:04:35.010725 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:35.010706    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="958c7031-f31a-4c34-9dd3-d01530719385" path="/var/lib/kubelet/pods/958c7031-f31a-4c34-9dd3-d01530719385/volumes"
Oct 30 14:04:35.133909 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:35.133865    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-310546-7f57cb9f95-5ghvp"
Oct 30 14:04:35.181721 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:35.181685    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b819faba-b9f4-488f-bfa0-4da40fb59313-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b819faba-b9f4-488f-bfa0-4da40fb59313-openshift-service-ca\") pod \"b819faba-b9f4-488f-bfa0-4da40fb59313\" (UID: \"b819faba-b9f4-488f-bfa0-4da40fb59313\") "
Oct 30 14:04:35.181925 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:35.181842    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b819faba-b9f4-488f-bfa0-4da40fb59313-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b819faba-b9f4-488f-bfa0-4da40fb59313-metrics-certs\") pod \"b819faba-b9f4-488f-bfa0-4da40fb59313\" (UID: \"b819faba-b9f4-488f-bfa0-4da40fb59313\") "
Oct 30 14:04:35.181925 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:35.181865    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b819faba-b9f4-488f-bfa0-4da40fb59313-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b819faba-b9f4-488f-bfa0-4da40fb59313-maas-api-tls\") pod \"b819faba-b9f4-488f-bfa0-4da40fb59313\" (UID: \"b819faba-b9f4-488f-bfa0-4da40fb59313\") "
Oct 30 14:04:35.181925 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:35.181898    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b819faba-b9f4-488f-bfa0-4da40fb59313-kube-api-access-pcsqw\" (UniqueName: \"kubernetes.io/projected/b819faba-b9f4-488f-bfa0-4da40fb59313-kube-api-access-pcsqw\") pod \"b819faba-b9f4-488f-bfa0-4da40fb59313\" (UID: \"b819faba-b9f4-488f-bfa0-4da40fb59313\") "
Oct 30 14:04:35.182174 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:35.182146    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b819faba-b9f4-488f-bfa0-4da40fb59313-openshift-service-ca" pod "b819faba-b9f4-488f-bfa0-4da40fb59313" (UID: "b819faba-b9f4-488f-bfa0-4da40fb59313"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:04:35.182495 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:35.182471    5093 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b819faba-b9f4-488f-bfa0-4da40fb59313-openshift-service-ca\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:04:35.184398 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:35.184340    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b819faba-b9f4-488f-bfa0-4da40fb59313-metrics-certs" pod "b819faba-b9f4-488f-bfa0-4da40fb59313" (UID: "b819faba-b9f4-488f-bfa0-4da40fb59313"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:04:35.184540 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:35.184500    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b819faba-b9f4-488f-bfa0-4da40fb59313-kube-api-access-pcsqw" pod "b819faba-b9f4-488f-bfa0-4da40fb59313" (UID: "b819faba-b9f4-488f-bfa0-4da40fb59313"). InnerVolumeSpecName "kube-api-access-pcsqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:04:35.185084 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:35.185055    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b819faba-b9f4-488f-bfa0-4da40fb59313-maas-api-tls" pod "b819faba-b9f4-488f-bfa0-4da40fb59313" (UID: "b819faba-b9f4-488f-bfa0-4da40fb59313"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:04:35.283577 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:35.283539    5093 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b819faba-b9f4-488f-bfa0-4da40fb59313-metrics-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:04:35.283577 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:35.283576    5093 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b819faba-b9f4-488f-bfa0-4da40fb59313-maas-api-tls\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:04:35.283871 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:35.283589    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcsqw\" (UniqueName: \"kubernetes.io/projected/b819faba-b9f4-488f-bfa0-4da40fb59313-kube-api-access-pcsqw\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:04:35.897031 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:35.896994    5093 generic.go:356] "Generic (PLEG): container finished" podID="b819faba-b9f4-488f-bfa0-4da40fb59313" containerID="58a1c5009b3015c501c0e183d602e2576f1bc53570e7616fbb66fcd1c8838838" exitCode=0
Oct 30 14:04:35.897530 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:35.897043    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-310546-7f57cb9f95-5ghvp"
Oct 30 14:04:35.897530 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:35.897088    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-310546-7f57cb9f95-5ghvp" event={"ID":"b819faba-b9f4-488f-bfa0-4da40fb59313","Type":"ContainerDied","Data":"58a1c5009b3015c501c0e183d602e2576f1bc53570e7616fbb66fcd1c8838838"}
Oct 30 14:04:35.897530 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:35.897167    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-310546-7f57cb9f95-5ghvp" event={"ID":"b819faba-b9f4-488f-bfa0-4da40fb59313","Type":"ContainerDied","Data":"8505c865ae1e0cd5aedda4e7b8ebc09c54d73f58889938b3585b8118549d8b67"}
Oct 30 14:04:35.897530 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:35.897198    5093 scope.go:122] "RemoveContainer" containerID="58a1c5009b3015c501c0e183d602e2576f1bc53570e7616fbb66fcd1c8838838"
Oct 30 14:04:35.909362 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:35.909339    5093 scope.go:122] "RemoveContainer" containerID="58a1c5009b3015c501c0e183d602e2576f1bc53570e7616fbb66fcd1c8838838"
Oct 30 14:04:35.909668 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:04:35.909641    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58a1c5009b3015c501c0e183d602e2576f1bc53570e7616fbb66fcd1c8838838\": container with ID starting with 58a1c5009b3015c501c0e183d602e2576f1bc53570e7616fbb66fcd1c8838838 not found: ID does not exist" containerID="58a1c5009b3015c501c0e183d602e2576f1bc53570e7616fbb66fcd1c8838838"
Oct 30 14:04:35.909749 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:35.909682    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58a1c5009b3015c501c0e183d602e2576f1bc53570e7616fbb66fcd1c8838838"} err="failed to get container status \"58a1c5009b3015c501c0e183d602e2576f1bc53570e7616fbb66fcd1c8838838\": rpc error: code = NotFound desc = could not find container \"58a1c5009b3015c501c0e183d602e2576f1bc53570e7616fbb66fcd1c8838838\": container with ID starting with 58a1c5009b3015c501c0e183d602e2576f1bc53570e7616fbb66fcd1c8838838 not found: ID does not exist"
Oct 30 14:04:35.978103 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:35.978033    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-310546-7f57cb9f95-5ghvp"]
Oct 30 14:04:35.991274 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:35.991235    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-310546-7f57cb9f95-5ghvp"]
Oct 30 14:04:37.000576 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:37.000536    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b819faba-b9f4-488f-bfa0-4da40fb59313" path="/var/lib/kubelet/pods/b819faba-b9f4-488f-bfa0-4da40fb59313/volumes"
Oct 30 14:04:51.168945 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:04:51.168901    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 14:04:54.597020 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:54.596983    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e3eb98-798ccfff95-kpvst"]
Oct 30 14:04:59.598260 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:04:59.598217    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e3eb98-798ccfff95-kpvst" podUID="cd554e69-a448-4d73-885a-d1ffcc724c56" containerName="payload-pre-processing" containerID="cri-o://e7a4ce73945cd8cc622d77f8046b94243f11a3d48b172fa45be6664297bb2ca6" gracePeriod=25
Oct 30 14:05:03.745711 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:05:03.745669    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e3eb98-798ccfff95-kpvst" podUID="cd554e69-a448-4d73-885a-d1ffcc724c56" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:05:13.745792 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:05:13.745682    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e3eb98-798ccfff95-kpvst" podUID="cd554e69-a448-4d73-885a-d1ffcc724c56" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:05:19.893824 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:05:19.893792    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e3eb98-798ccfff95-kpvst"
Oct 30 14:05:19.996634 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:05:19.996603    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd554e69-a448-4d73-885a-d1ffcc724c56-kube-api-access-dk997\" (UniqueName: \"kubernetes.io/projected/cd554e69-a448-4d73-885a-d1ffcc724c56-kube-api-access-dk997\") pod \"cd554e69-a448-4d73-885a-d1ffcc724c56\" (UID: \"cd554e69-a448-4d73-885a-d1ffcc724c56\") "
Oct 30 14:05:19.996817 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:05:19.996653    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd554e69-a448-4d73-885a-d1ffcc724c56-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd554e69-a448-4d73-885a-d1ffcc724c56-plugins-config-volume\") pod \"cd554e69-a448-4d73-885a-d1ffcc724c56\" (UID: \"cd554e69-a448-4d73-885a-d1ffcc724c56\") "
Oct 30 14:05:19.997375 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:05:19.997350    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd554e69-a448-4d73-885a-d1ffcc724c56-plugins-config-volume" pod "cd554e69-a448-4d73-885a-d1ffcc724c56" (UID: "cd554e69-a448-4d73-885a-d1ffcc724c56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:05:19.999560 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:05:19.999525    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd554e69-a448-4d73-885a-d1ffcc724c56-kube-api-access-dk997" pod "cd554e69-a448-4d73-885a-d1ffcc724c56" (UID: "cd554e69-a448-4d73-885a-d1ffcc724c56"). InnerVolumeSpecName "kube-api-access-dk997". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:05:20.098856 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:05:20.098808    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk997\" (UniqueName: \"kubernetes.io/projected/cd554e69-a448-4d73-885a-d1ffcc724c56-kube-api-access-dk997\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:05:20.098856 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:05:20.098844    5093 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd554e69-a448-4d73-885a-d1ffcc724c56-plugins-config-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:05:20.294353 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:05:20.294314    5093 generic.go:356] "Generic (PLEG): container finished" podID="cd554e69-a448-4d73-885a-d1ffcc724c56" containerID="e7a4ce73945cd8cc622d77f8046b94243f11a3d48b172fa45be6664297bb2ca6" exitCode=0
Oct 30 14:05:20.294547 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:05:20.294362    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e3eb98-798ccfff95-kpvst"
Oct 30 14:05:20.294547 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:05:20.294394    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e3eb98-798ccfff95-kpvst" event={"ID":"cd554e69-a448-4d73-885a-d1ffcc724c56","Type":"ContainerDied","Data":"e7a4ce73945cd8cc622d77f8046b94243f11a3d48b172fa45be6664297bb2ca6"}
Oct 30 14:05:20.294547 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:05:20.294454    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e3eb98-798ccfff95-kpvst" event={"ID":"cd554e69-a448-4d73-885a-d1ffcc724c56","Type":"ContainerDied","Data":"5e0957dcf07766b9e12fe9a163d82e569639dd84b8e145baa2bb71839b828309"}
Oct 30 14:05:20.294547 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:05:20.294484    5093 scope.go:122] "RemoveContainer" containerID="e7a4ce73945cd8cc622d77f8046b94243f11a3d48b172fa45be6664297bb2ca6"
Oct 30 14:05:20.307423 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:05:20.307399    5093 scope.go:122] "RemoveContainer" containerID="e7a4ce73945cd8cc622d77f8046b94243f11a3d48b172fa45be6664297bb2ca6"
Oct 30 14:05:20.307733 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:05:20.307708    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7a4ce73945cd8cc622d77f8046b94243f11a3d48b172fa45be6664297bb2ca6\": container with ID starting with e7a4ce73945cd8cc622d77f8046b94243f11a3d48b172fa45be6664297bb2ca6 not found: ID does not exist" containerID="e7a4ce73945cd8cc622d77f8046b94243f11a3d48b172fa45be6664297bb2ca6"
Oct 30 14:05:20.307816 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:05:20.307745    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7a4ce73945cd8cc622d77f8046b94243f11a3d48b172fa45be6664297bb2ca6"} err="failed to get container status \"e7a4ce73945cd8cc622d77f8046b94243f11a3d48b172fa45be6664297bb2ca6\": rpc error: code = NotFound desc = could not find container \"e7a4ce73945cd8cc622d77f8046b94243f11a3d48b172fa45be6664297bb2ca6\": container with ID starting with e7a4ce73945cd8cc622d77f8046b94243f11a3d48b172fa45be6664297bb2ca6 not found: ID does not exist"
Oct 30 14:05:20.353902 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:05:20.353869    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e3eb98-798ccfff95-kpvst"]
Oct 30 14:05:20.354620 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:05:20.354600    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e3eb98-798ccfff95-kpvst"]
Oct 30 14:05:21.002536 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:05:21.002502    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd554e69-a448-4d73-885a-d1ffcc724c56" path="/var/lib/kubelet/pods/cd554e69-a448-4d73-885a-d1ffcc724c56/volumes"
Oct 30 14:05:21.172039 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:05:21.171996    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 14:05:33.447526 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:05:33.447487    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3eb98-7789bfbbbc-c4n45"]
Oct 30 14:05:33.448355 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:05:33.448316    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3eb98-7789bfbbbc-c4n45" podUID="c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd" containerName="maas-api" containerID="cri-o://d9553bd1508bc27465bdbf17d7c047329e29dd0f1bcc9cd832de0c8cbc1a5cf3" gracePeriod=30
Oct 30 14:05:33.809289 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:05:33.809250    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3eb98-7789bfbbbc-c4n45"
Oct 30 14:05:33.938193 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:05:33.935712    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd-metrics-certs\") pod \"c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd\" (UID: \"c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd\") "
Oct 30 14:05:33.938193 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:05:33.935901    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd-openshift-service-ca\") pod \"c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd\" (UID: \"c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd\") "
Oct 30 14:05:33.938193 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:05:33.935972    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd-maas-api-tls\") pod \"c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd\" (UID: \"c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd\") "
Oct 30 14:05:33.938193 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:05:33.936034    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd-kube-api-access-nmghd\" (UniqueName: \"kubernetes.io/projected/c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd-kube-api-access-nmghd\") pod \"c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd\" (UID: \"c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd\") "
Oct 30 14:05:33.938193 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:05:33.937345    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd-openshift-service-ca" pod "c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd" (UID: "c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:05:33.955568 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:05:33.955145    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd-maas-api-tls" pod "c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd" (UID: "c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:05:33.962404 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:05:33.962364    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd-metrics-certs" pod "c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd" (UID: "c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:05:33.978271 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:05:33.970567    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd-kube-api-access-nmghd" pod "c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd" (UID: "c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd"). InnerVolumeSpecName "kube-api-access-nmghd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:05:34.038079 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:05:34.037931    5093 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd-maas-api-tls\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:05:34.038079 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:05:34.038012    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmghd\" (UniqueName: \"kubernetes.io/projected/c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd-kube-api-access-nmghd\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:05:34.038079 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:05:34.038051    5093 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd-metrics-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:05:34.038404 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:05:34.038065    5093 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd-openshift-service-ca\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:05:34.429076 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:05:34.429039    5093 generic.go:356] "Generic (PLEG): container finished" podID="c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd" containerID="d9553bd1508bc27465bdbf17d7c047329e29dd0f1bcc9cd832de0c8cbc1a5cf3" exitCode=0
Oct 30 14:05:34.429286 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:05:34.429094    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3eb98-7789bfbbbc-c4n45"
Oct 30 14:05:34.429286 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:05:34.429119    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3eb98-7789bfbbbc-c4n45" event={"ID":"c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd","Type":"ContainerDied","Data":"d9553bd1508bc27465bdbf17d7c047329e29dd0f1bcc9cd832de0c8cbc1a5cf3"}
Oct 30 14:05:34.429286 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:05:34.429215    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3eb98-7789bfbbbc-c4n45" event={"ID":"c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd","Type":"ContainerDied","Data":"d40fd3e2fbdd55d99981af0b0c6ceee53b7ebb9fdb8125997088f85ae38ff4d7"}
Oct 30 14:05:34.429286 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:05:34.429238    5093 scope.go:122] "RemoveContainer" containerID="d9553bd1508bc27465bdbf17d7c047329e29dd0f1bcc9cd832de0c8cbc1a5cf3"
Oct 30 14:05:34.443906 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:05:34.443872    5093 scope.go:122] "RemoveContainer" containerID="d9553bd1508bc27465bdbf17d7c047329e29dd0f1bcc9cd832de0c8cbc1a5cf3"
Oct 30 14:05:34.444329 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:05:34.444301    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9553bd1508bc27465bdbf17d7c047329e29dd0f1bcc9cd832de0c8cbc1a5cf3\": container with ID starting with d9553bd1508bc27465bdbf17d7c047329e29dd0f1bcc9cd832de0c8cbc1a5cf3 not found: ID does not exist" containerID="d9553bd1508bc27465bdbf17d7c047329e29dd0f1bcc9cd832de0c8cbc1a5cf3"
Oct 30 14:05:34.444452 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:05:34.444340    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9553bd1508bc27465bdbf17d7c047329e29dd0f1bcc9cd832de0c8cbc1a5cf3"} err="failed to get container status \"d9553bd1508bc27465bdbf17d7c047329e29dd0f1bcc9cd832de0c8cbc1a5cf3\": rpc error: code = NotFound desc = could not find container \"d9553bd1508bc27465bdbf17d7c047329e29dd0f1bcc9cd832de0c8cbc1a5cf3\": container with ID starting with d9553bd1508bc27465bdbf17d7c047329e29dd0f1bcc9cd832de0c8cbc1a5cf3 not found: ID does not exist"
Oct 30 14:05:34.487053 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:05:34.487015    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3eb98-7789bfbbbc-c4n45"]
Oct 30 14:05:34.500222 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:05:34.500189    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3eb98-7789bfbbbc-c4n45"]
Oct 30 14:05:35.000893 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:05:35.000847    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd" path="/var/lib/kubelet/pods/c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd/volumes"
Oct 30 14:05:51.168963 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:05:51.168916    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 14:05:52.451413 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:05:52.451372    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-cab6c5-555d87bc58-6slkr"]
Oct 30 14:05:52.478191 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:05:52.478153    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cab6c5-866cf94959-h48bz"]
Oct 30 14:05:57.452179 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:05:57.452079    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cab6c5-555d87bc58-6slkr" podUID="6bcb5af7-5a52-48b1-88fa-e5b1359baced" containerName="payload-processing" containerID="cri-o://e8049095d31a5da478724d2e649f76e7d86f1da5f89705b70b2d03807df994ff" gracePeriod=25
Oct 30 14:05:57.479093 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:05:57.479042    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cab6c5-866cf94959-h48bz" podUID="70b59b98-9888-4d17-8695-e16a21f08fb0" containerName="payload-pre-processing" containerID="cri-o://86322fd73affd84416a817e2af7955e48a56583913a34f94c5f4ea82bee54714" gracePeriod=25
Oct 30 14:05:59.529231 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:05:59.529175    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cab6c5-866cf94959-h48bz" podUID="70b59b98-9888-4d17-8695-e16a21f08fb0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:05:59.529700 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:05:59.529242    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cab6c5-555d87bc58-6slkr" podUID="6bcb5af7-5a52-48b1-88fa-e5b1359baced" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:06:09.529455 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:09.529411    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cab6c5-866cf94959-h48bz" podUID="70b59b98-9888-4d17-8695-e16a21f08fb0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:06:09.530038 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:09.529548    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cab6c5-555d87bc58-6slkr" podUID="6bcb5af7-5a52-48b1-88fa-e5b1359baced" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:06:17.811477 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:17.811442    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cab6c5-555d87bc58-6slkr"
Oct 30 14:06:17.820751 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:17.820722    5093 generic.go:356] "Generic (PLEG): container finished" podID="70b59b98-9888-4d17-8695-e16a21f08fb0" containerID="86322fd73affd84416a817e2af7955e48a56583913a34f94c5f4ea82bee54714" exitCode=0
Oct 30 14:06:17.820894 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:17.820795    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cab6c5-866cf94959-h48bz" event={"ID":"70b59b98-9888-4d17-8695-e16a21f08fb0","Type":"ContainerDied","Data":"86322fd73affd84416a817e2af7955e48a56583913a34f94c5f4ea82bee54714"}
Oct 30 14:06:17.822351 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:17.822328    5093 generic.go:356] "Generic (PLEG): container finished" podID="6bcb5af7-5a52-48b1-88fa-e5b1359baced" containerID="e8049095d31a5da478724d2e649f76e7d86f1da5f89705b70b2d03807df994ff" exitCode=0
Oct 30 14:06:17.822477 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:17.822363    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cab6c5-555d87bc58-6slkr" event={"ID":"6bcb5af7-5a52-48b1-88fa-e5b1359baced","Type":"ContainerDied","Data":"e8049095d31a5da478724d2e649f76e7d86f1da5f89705b70b2d03807df994ff"}
Oct 30 14:06:17.822477 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:17.822374    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cab6c5-555d87bc58-6slkr"
Oct 30 14:06:17.822477 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:17.822405    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cab6c5-555d87bc58-6slkr" event={"ID":"6bcb5af7-5a52-48b1-88fa-e5b1359baced","Type":"ContainerDied","Data":"9b891c71837a2f528aad02919b8e27c31a3cc98b1f3ccb102d99de969a0c7d3f"}
Oct 30 14:06:17.822477 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:17.822430    5093 scope.go:122] "RemoveContainer" containerID="e8049095d31a5da478724d2e649f76e7d86f1da5f89705b70b2d03807df994ff"
Oct 30 14:06:17.833884 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:17.833857    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cab6c5-866cf94959-h48bz"
Oct 30 14:06:17.836272 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:17.836243    5093 scope.go:122] "RemoveContainer" containerID="e8049095d31a5da478724d2e649f76e7d86f1da5f89705b70b2d03807df994ff"
Oct 30 14:06:17.836596 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:06:17.836569    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8049095d31a5da478724d2e649f76e7d86f1da5f89705b70b2d03807df994ff\": container with ID starting with e8049095d31a5da478724d2e649f76e7d86f1da5f89705b70b2d03807df994ff not found: ID does not exist" containerID="e8049095d31a5da478724d2e649f76e7d86f1da5f89705b70b2d03807df994ff"
Oct 30 14:06:17.836683 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:17.836611    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8049095d31a5da478724d2e649f76e7d86f1da5f89705b70b2d03807df994ff"} err="failed to get container status \"e8049095d31a5da478724d2e649f76e7d86f1da5f89705b70b2d03807df994ff\": rpc error: code = NotFound desc = could not find container \"e8049095d31a5da478724d2e649f76e7d86f1da5f89705b70b2d03807df994ff\": container with ID starting with e8049095d31a5da478724d2e649f76e7d86f1da5f89705b70b2d03807df994ff not found: ID does not exist"
Oct 30 14:06:17.973251 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:17.973213    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bcb5af7-5a52-48b1-88fa-e5b1359baced-kube-api-access-fl2pg\" (UniqueName: \"kubernetes.io/projected/6bcb5af7-5a52-48b1-88fa-e5b1359baced-kube-api-access-fl2pg\") pod \"6bcb5af7-5a52-48b1-88fa-e5b1359baced\" (UID: \"6bcb5af7-5a52-48b1-88fa-e5b1359baced\") "
Oct 30 14:06:17.973469 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:17.973287    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bcb5af7-5a52-48b1-88fa-e5b1359baced-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bcb5af7-5a52-48b1-88fa-e5b1359baced-plugins-config-volume\") pod \"6bcb5af7-5a52-48b1-88fa-e5b1359baced\" (UID: \"6bcb5af7-5a52-48b1-88fa-e5b1359baced\") "
Oct 30 14:06:17.973469 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:17.973330    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70b59b98-9888-4d17-8695-e16a21f08fb0-kube-api-access-wnfcc\" (UniqueName: \"kubernetes.io/projected/70b59b98-9888-4d17-8695-e16a21f08fb0-kube-api-access-wnfcc\") pod \"70b59b98-9888-4d17-8695-e16a21f08fb0\" (UID: \"70b59b98-9888-4d17-8695-e16a21f08fb0\") "
Oct 30 14:06:17.973469 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:17.973387    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70b59b98-9888-4d17-8695-e16a21f08fb0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70b59b98-9888-4d17-8695-e16a21f08fb0-plugins-config-volume\") pod \"70b59b98-9888-4d17-8695-e16a21f08fb0\" (UID: \"70b59b98-9888-4d17-8695-e16a21f08fb0\") "
Oct 30 14:06:17.973834 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:17.973803    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bcb5af7-5a52-48b1-88fa-e5b1359baced-plugins-config-volume" pod "6bcb5af7-5a52-48b1-88fa-e5b1359baced" (UID: "6bcb5af7-5a52-48b1-88fa-e5b1359baced"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:06:17.973981 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:17.973875    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70b59b98-9888-4d17-8695-e16a21f08fb0-plugins-config-volume" pod "70b59b98-9888-4d17-8695-e16a21f08fb0" (UID: "70b59b98-9888-4d17-8695-e16a21f08fb0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:06:17.974098 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:17.974076    5093 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bcb5af7-5a52-48b1-88fa-e5b1359baced-plugins-config-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:06:17.974175 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:17.974103    5093 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70b59b98-9888-4d17-8695-e16a21f08fb0-plugins-config-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:06:17.975683 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:17.975660    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70b59b98-9888-4d17-8695-e16a21f08fb0-kube-api-access-wnfcc" pod "70b59b98-9888-4d17-8695-e16a21f08fb0" (UID: "70b59b98-9888-4d17-8695-e16a21f08fb0"). InnerVolumeSpecName "kube-api-access-wnfcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:06:17.975992 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:17.975962    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bcb5af7-5a52-48b1-88fa-e5b1359baced-kube-api-access-fl2pg" pod "6bcb5af7-5a52-48b1-88fa-e5b1359baced" (UID: "6bcb5af7-5a52-48b1-88fa-e5b1359baced"). InnerVolumeSpecName "kube-api-access-fl2pg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:06:18.075440 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:18.075305    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl2pg\" (UniqueName: \"kubernetes.io/projected/6bcb5af7-5a52-48b1-88fa-e5b1359baced-kube-api-access-fl2pg\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:06:18.075440 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:18.075363    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnfcc\" (UniqueName: \"kubernetes.io/projected/70b59b98-9888-4d17-8695-e16a21f08fb0-kube-api-access-wnfcc\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:06:18.171250 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:18.171199    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-cab6c5-555d87bc58-6slkr"]
Oct 30 14:06:18.177548 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:18.177514    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-cab6c5-555d87bc58-6slkr"]
Oct 30 14:06:18.832479 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:18.832386    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cab6c5-866cf94959-h48bz"
Oct 30 14:06:18.832479 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:18.832430    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cab6c5-866cf94959-h48bz" event={"ID":"70b59b98-9888-4d17-8695-e16a21f08fb0","Type":"ContainerDied","Data":"b6350e9ee987d2cb4460ef5f4c196fe95bc4e0bd488043ed4b21d54d9d3bd3c5"}
Oct 30 14:06:18.833025 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:18.832507    5093 scope.go:122] "RemoveContainer" containerID="86322fd73affd84416a817e2af7955e48a56583913a34f94c5f4ea82bee54714"
Oct 30 14:06:18.902518 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:18.902482    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cab6c5-866cf94959-h48bz"]
Oct 30 14:06:18.915712 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:18.915676    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cab6c5-866cf94959-h48bz"]
Oct 30 14:06:19.001629 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:19.001591    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bcb5af7-5a52-48b1-88fa-e5b1359baced" path="/var/lib/kubelet/pods/6bcb5af7-5a52-48b1-88fa-e5b1359baced/volumes"
Oct 30 14:06:19.002081 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:19.002065    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70b59b98-9888-4d17-8695-e16a21f08fb0" path="/var/lib/kubelet/pods/70b59b98-9888-4d17-8695-e16a21f08fb0/volumes"
Oct 30 14:06:21.170773 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:06:21.170734    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 14:06:28.713067 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:28.713029    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-bd2fed-68ff7bd4b6-d9tt2"]
Oct 30 14:06:30.571726 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:30.571685    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cab6c5-8559b4f669-ktcqz"]
Oct 30 14:06:30.572245 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:30.571909    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cab6c5-8559b4f669-ktcqz" podUID="14d0074c-a3ab-4ec3-b8e8-95b4681260fd" containerName="maas-api" containerID="cri-o://f9e314a6be63c6c95b6952d98b38ff6363c2e3e9c27dde69d356a911807d94ca" gracePeriod=30
Oct 30 14:06:30.891461 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:30.891417    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cab6c5-8559b4f669-ktcqz"
Oct 30 14:06:30.943932 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:30.943893    5093 generic.go:356] "Generic (PLEG): container finished" podID="14d0074c-a3ab-4ec3-b8e8-95b4681260fd" containerID="f9e314a6be63c6c95b6952d98b38ff6363c2e3e9c27dde69d356a911807d94ca" exitCode=0
Oct 30 14:06:30.944815 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:30.943956    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cab6c5-8559b4f669-ktcqz"
Oct 30 14:06:30.944815 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:30.943983    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cab6c5-8559b4f669-ktcqz" event={"ID":"14d0074c-a3ab-4ec3-b8e8-95b4681260fd","Type":"ContainerDied","Data":"f9e314a6be63c6c95b6952d98b38ff6363c2e3e9c27dde69d356a911807d94ca"}
Oct 30 14:06:30.944815 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:30.944058    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cab6c5-8559b4f669-ktcqz" event={"ID":"14d0074c-a3ab-4ec3-b8e8-95b4681260fd","Type":"ContainerDied","Data":"0873e79b6406e53887f57f81c3c162d2771eb8a8315af97a3ae33ce67f451756"}
Oct 30 14:06:30.944815 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:30.944092    5093 scope.go:122] "RemoveContainer" containerID="f9e314a6be63c6c95b6952d98b38ff6363c2e3e9c27dde69d356a911807d94ca"
Oct 30 14:06:30.967379 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:30.967352    5093 scope.go:122] "RemoveContainer" containerID="f9e314a6be63c6c95b6952d98b38ff6363c2e3e9c27dde69d356a911807d94ca"
Oct 30 14:06:30.967802 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:06:30.967771    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9e314a6be63c6c95b6952d98b38ff6363c2e3e9c27dde69d356a911807d94ca\": container with ID starting with f9e314a6be63c6c95b6952d98b38ff6363c2e3e9c27dde69d356a911807d94ca not found: ID does not exist" containerID="f9e314a6be63c6c95b6952d98b38ff6363c2e3e9c27dde69d356a911807d94ca"
Oct 30 14:06:30.967932 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:30.967818    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9e314a6be63c6c95b6952d98b38ff6363c2e3e9c27dde69d356a911807d94ca"} err="failed to get container status \"f9e314a6be63c6c95b6952d98b38ff6363c2e3e9c27dde69d356a911807d94ca\": rpc error: code = NotFound desc = could not find container \"f9e314a6be63c6c95b6952d98b38ff6363c2e3e9c27dde69d356a911807d94ca\": container with ID starting with f9e314a6be63c6c95b6952d98b38ff6363c2e3e9c27dde69d356a911807d94ca not found: ID does not exist"
Oct 30 14:06:30.974601 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:30.974564    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14d0074c-a3ab-4ec3-b8e8-95b4681260fd-kube-api-access-slq69\" (UniqueName: \"kubernetes.io/projected/14d0074c-a3ab-4ec3-b8e8-95b4681260fd-kube-api-access-slq69\") pod \"14d0074c-a3ab-4ec3-b8e8-95b4681260fd\" (UID: \"14d0074c-a3ab-4ec3-b8e8-95b4681260fd\") "
Oct 30 14:06:30.974783 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:30.974737    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14d0074c-a3ab-4ec3-b8e8-95b4681260fd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14d0074c-a3ab-4ec3-b8e8-95b4681260fd-openshift-service-ca\") pod \"14d0074c-a3ab-4ec3-b8e8-95b4681260fd\" (UID: \"14d0074c-a3ab-4ec3-b8e8-95b4681260fd\") "
Oct 30 14:06:30.974929 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:30.974911    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14d0074c-a3ab-4ec3-b8e8-95b4681260fd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14d0074c-a3ab-4ec3-b8e8-95b4681260fd-maas-api-tls\") pod \"14d0074c-a3ab-4ec3-b8e8-95b4681260fd\" (UID: \"14d0074c-a3ab-4ec3-b8e8-95b4681260fd\") "
Oct 30 14:06:30.975006 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:30.974990    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14d0074c-a3ab-4ec3-b8e8-95b4681260fd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/14d0074c-a3ab-4ec3-b8e8-95b4681260fd-metrics-certs\") pod \"14d0074c-a3ab-4ec3-b8e8-95b4681260fd\" (UID: \"14d0074c-a3ab-4ec3-b8e8-95b4681260fd\") "
Oct 30 14:06:30.975362 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:30.975305    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14d0074c-a3ab-4ec3-b8e8-95b4681260fd-openshift-service-ca" pod "14d0074c-a3ab-4ec3-b8e8-95b4681260fd" (UID: "14d0074c-a3ab-4ec3-b8e8-95b4681260fd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:06:30.975735 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:30.975709    5093 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14d0074c-a3ab-4ec3-b8e8-95b4681260fd-openshift-service-ca\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:06:30.977122 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:30.977089    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14d0074c-a3ab-4ec3-b8e8-95b4681260fd-kube-api-access-slq69" pod "14d0074c-a3ab-4ec3-b8e8-95b4681260fd" (UID: "14d0074c-a3ab-4ec3-b8e8-95b4681260fd"). InnerVolumeSpecName "kube-api-access-slq69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:06:30.977450 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:30.977420    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14d0074c-a3ab-4ec3-b8e8-95b4681260fd-maas-api-tls" pod "14d0074c-a3ab-4ec3-b8e8-95b4681260fd" (UID: "14d0074c-a3ab-4ec3-b8e8-95b4681260fd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:06:30.977613 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:30.977595    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14d0074c-a3ab-4ec3-b8e8-95b4681260fd-metrics-certs" pod "14d0074c-a3ab-4ec3-b8e8-95b4681260fd" (UID: "14d0074c-a3ab-4ec3-b8e8-95b4681260fd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:06:31.077000 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:31.076961    5093 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14d0074c-a3ab-4ec3-b8e8-95b4681260fd-maas-api-tls\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:06:31.077000 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:31.076998    5093 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14d0074c-a3ab-4ec3-b8e8-95b4681260fd-metrics-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:06:31.077256 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:31.077013    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slq69\" (UniqueName: \"kubernetes.io/projected/14d0074c-a3ab-4ec3-b8e8-95b4681260fd-kube-api-access-slq69\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:06:31.286996 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:31.286954    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cab6c5-8559b4f669-ktcqz"]
Oct 30 14:06:31.299959 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:31.299914    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cab6c5-8559b4f669-ktcqz"]
Oct 30 14:06:33.000452 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:33.000415    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14d0074c-a3ab-4ec3-b8e8-95b4681260fd" path="/var/lib/kubelet/pods/14d0074c-a3ab-4ec3-b8e8-95b4681260fd/volumes"
Oct 30 14:06:33.713350 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:33.713295    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-bd2fed-68ff7bd4b6-d9tt2" podUID="64b19bcc-57c8-4838-a94c-5b918d4da2e2" containerName="payload-processing" containerID="cri-o://8a527cd176c5ede8c33363022a30d2134b4b009b09d12b4fa5e36779313431da" gracePeriod=25
Oct 30 14:06:37.922589 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:37.922547    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-bd2fed-68ff7bd4b6-d9tt2" podUID="64b19bcc-57c8-4838-a94c-5b918d4da2e2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:06:39.174830 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:39.174787    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp"]
Oct 30 14:06:39.175358 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:39.175028    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp" podUID="cc75f0e6-64c6-4fde-9476-3859350da986" containerName="istio-proxy" containerID="cri-o://9826755f30199716a4db91c54e8186873daa229ff6f3eb83d664150033f7454f" gracePeriod=30
Oct 30 14:06:41.435391 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:41.435286    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd2fed-67b8dc8f65-v5kx7"]
Oct 30 14:06:41.435915 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:41.435547    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd2fed-67b8dc8f65-v5kx7" podUID="00f2b03f-864c-4e03-9144-b949bced2ba9" containerName="maas-api" containerID="cri-o://673eee8ca1cbb0b5404386eb25583c30c116052ec63261f8991aac7296a4f7f0" gracePeriod=30
Oct 30 14:06:41.788847 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:41.788816    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd2fed-67b8dc8f65-v5kx7"
Oct 30 14:06:41.875854 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:41.874288    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00f2b03f-864c-4e03-9144-b949bced2ba9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00f2b03f-864c-4e03-9144-b949bced2ba9-openshift-service-ca\") pod \"00f2b03f-864c-4e03-9144-b949bced2ba9\" (UID: \"00f2b03f-864c-4e03-9144-b949bced2ba9\") "
Oct 30 14:06:41.875854 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:41.874459    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00f2b03f-864c-4e03-9144-b949bced2ba9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/00f2b03f-864c-4e03-9144-b949bced2ba9-metrics-certs\") pod \"00f2b03f-864c-4e03-9144-b949bced2ba9\" (UID: \"00f2b03f-864c-4e03-9144-b949bced2ba9\") "
Oct 30 14:06:41.875854 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:41.874606    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00f2b03f-864c-4e03-9144-b949bced2ba9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00f2b03f-864c-4e03-9144-b949bced2ba9-maas-api-tls\") pod \"00f2b03f-864c-4e03-9144-b949bced2ba9\" (UID: \"00f2b03f-864c-4e03-9144-b949bced2ba9\") "
Oct 30 14:06:41.875854 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:41.874674    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00f2b03f-864c-4e03-9144-b949bced2ba9-kube-api-access-vmtbf\" (UniqueName: \"kubernetes.io/projected/00f2b03f-864c-4e03-9144-b949bced2ba9-kube-api-access-vmtbf\") pod \"00f2b03f-864c-4e03-9144-b949bced2ba9\" (UID: \"00f2b03f-864c-4e03-9144-b949bced2ba9\") "
Oct 30 14:06:41.876498 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:41.876234    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00f2b03f-864c-4e03-9144-b949bced2ba9-openshift-service-ca" pod "00f2b03f-864c-4e03-9144-b949bced2ba9" (UID: "00f2b03f-864c-4e03-9144-b949bced2ba9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:06:41.902681 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:41.902189    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00f2b03f-864c-4e03-9144-b949bced2ba9-kube-api-access-vmtbf" pod "00f2b03f-864c-4e03-9144-b949bced2ba9" (UID: "00f2b03f-864c-4e03-9144-b949bced2ba9"). InnerVolumeSpecName "kube-api-access-vmtbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:06:41.902681 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:41.902309    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00f2b03f-864c-4e03-9144-b949bced2ba9-maas-api-tls" pod "00f2b03f-864c-4e03-9144-b949bced2ba9" (UID: "00f2b03f-864c-4e03-9144-b949bced2ba9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:06:41.925519 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:41.925473    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00f2b03f-864c-4e03-9144-b949bced2ba9-metrics-certs" pod "00f2b03f-864c-4e03-9144-b949bced2ba9" (UID: "00f2b03f-864c-4e03-9144-b949bced2ba9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:06:41.976742 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:41.976695    5093 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00f2b03f-864c-4e03-9144-b949bced2ba9-openshift-service-ca\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:06:41.976742 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:41.976737    5093 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00f2b03f-864c-4e03-9144-b949bced2ba9-metrics-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:06:41.976742 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:41.976751    5093 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00f2b03f-864c-4e03-9144-b949bced2ba9-maas-api-tls\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:06:41.977070 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:41.976766    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmtbf\" (UniqueName: \"kubernetes.io/projected/00f2b03f-864c-4e03-9144-b949bced2ba9-kube-api-access-vmtbf\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:06:42.051882 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.051789    5093 generic.go:356] "Generic (PLEG): container finished" podID="00f2b03f-864c-4e03-9144-b949bced2ba9" containerID="673eee8ca1cbb0b5404386eb25583c30c116052ec63261f8991aac7296a4f7f0" exitCode=0
Oct 30 14:06:42.051882 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.051842    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd2fed-67b8dc8f65-v5kx7"
Oct 30 14:06:42.051882 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.051868    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd2fed-67b8dc8f65-v5kx7" event={"ID":"00f2b03f-864c-4e03-9144-b949bced2ba9","Type":"ContainerDied","Data":"673eee8ca1cbb0b5404386eb25583c30c116052ec63261f8991aac7296a4f7f0"}
Oct 30 14:06:42.052237 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.051929    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd2fed-67b8dc8f65-v5kx7" event={"ID":"00f2b03f-864c-4e03-9144-b949bced2ba9","Type":"ContainerDied","Data":"ef56e186ea5b1d1612d556b48f438904472c7e58ee93cb0605e232308f186188"}
Oct 30 14:06:42.052237 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.051951    5093 scope.go:122] "RemoveContainer" containerID="673eee8ca1cbb0b5404386eb25583c30c116052ec63261f8991aac7296a4f7f0"
Oct 30 14:06:42.070054 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.070022    5093 scope.go:122] "RemoveContainer" containerID="673eee8ca1cbb0b5404386eb25583c30c116052ec63261f8991aac7296a4f7f0"
Oct 30 14:06:42.070495 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:06:42.070464    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"673eee8ca1cbb0b5404386eb25583c30c116052ec63261f8991aac7296a4f7f0\": container with ID starting with 673eee8ca1cbb0b5404386eb25583c30c116052ec63261f8991aac7296a4f7f0 not found: ID does not exist" containerID="673eee8ca1cbb0b5404386eb25583c30c116052ec63261f8991aac7296a4f7f0"
Oct 30 14:06:42.070616 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.070506    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"673eee8ca1cbb0b5404386eb25583c30c116052ec63261f8991aac7296a4f7f0"} err="failed to get container status \"673eee8ca1cbb0b5404386eb25583c30c116052ec63261f8991aac7296a4f7f0\": rpc error: code = NotFound desc = could not find container \"673eee8ca1cbb0b5404386eb25583c30c116052ec63261f8991aac7296a4f7f0\": container with ID starting with 673eee8ca1cbb0b5404386eb25583c30c116052ec63261f8991aac7296a4f7f0 not found: ID does not exist"
Oct 30 14:06:42.111417 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.111366    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd2fed-67b8dc8f65-v5kx7"]
Oct 30 14:06:42.116784 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.116753    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd2fed-67b8dc8f65-v5kx7"]
Oct 30 14:06:42.722986 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.722905    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-51dc9a/rate-test-model-51dc9a-kserve-69b7d876df-gqk6q"]
Oct 30 14:06:42.723665 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.723650    5093 cpu_manager.go:414] "containerMap: removing container" podUID="b819faba-b9f4-488f-bfa0-4da40fb59313" containerName="maas-api"
Oct 30 14:06:42.723720 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.723668    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b819faba-b9f4-488f-bfa0-4da40fb59313" containerName="maas-api"
Oct 30 14:06:42.723720 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.723679    5093 cpu_manager.go:414] "containerMap: removing container" podUID="b676aeef-6a09-4e04-929d-6388b8034eac" containerName="maas-api"
Oct 30 14:06:42.723720 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.723684    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b676aeef-6a09-4e04-929d-6388b8034eac" containerName="maas-api"
Oct 30 14:06:42.723720 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.723692    5093 cpu_manager.go:414] "containerMap: removing container" podUID="70b59b98-9888-4d17-8695-e16a21f08fb0" containerName="payload-pre-processing"
Oct 30 14:06:42.723720 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.723698    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70b59b98-9888-4d17-8695-e16a21f08fb0" containerName="payload-pre-processing"
Oct 30 14:06:42.723720 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.723709    5093 cpu_manager.go:414] "containerMap: removing container" podUID="6bcb5af7-5a52-48b1-88fa-e5b1359baced" containerName="payload-processing"
Oct 30 14:06:42.723720 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.723714    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bcb5af7-5a52-48b1-88fa-e5b1359baced" containerName="payload-processing"
Oct 30 14:06:42.723951 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.723735    5093 cpu_manager.go:414] "containerMap: removing container" podUID="c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd" containerName="maas-api"
Oct 30 14:06:42.723951 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.723740    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd" containerName="maas-api"
Oct 30 14:06:42.723951 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.723752    5093 cpu_manager.go:414] "containerMap: removing container" podUID="63b21cf6-7b3b-48b6-88a1-d03e804d33fa" containerName="payload-pre-processing"
Oct 30 14:06:42.723951 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.723757    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63b21cf6-7b3b-48b6-88a1-d03e804d33fa" containerName="payload-pre-processing"
Oct 30 14:06:42.723951 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.723777    5093 cpu_manager.go:414] "containerMap: removing container" podUID="958c7031-f31a-4c34-9dd3-d01530719385" containerName="payload-processing"
Oct 30 14:06:42.723951 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.723781    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="958c7031-f31a-4c34-9dd3-d01530719385" containerName="payload-processing"
Oct 30 14:06:42.723951 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.723789    5093 cpu_manager.go:414] "containerMap: removing container" podUID="00f2b03f-864c-4e03-9144-b949bced2ba9" containerName="maas-api"
Oct 30 14:06:42.723951 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.723794    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00f2b03f-864c-4e03-9144-b949bced2ba9" containerName="maas-api"
Oct 30 14:06:42.723951 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.723802    5093 cpu_manager.go:414] "containerMap: removing container" podUID="cd554e69-a448-4d73-885a-d1ffcc724c56" containerName="payload-pre-processing"
Oct 30 14:06:42.723951 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.723806    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd554e69-a448-4d73-885a-d1ffcc724c56" containerName="payload-pre-processing"
Oct 30 14:06:42.723951 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.723818    5093 cpu_manager.go:414] "containerMap: removing container" podUID="14d0074c-a3ab-4ec3-b8e8-95b4681260fd" containerName="maas-api"
Oct 30 14:06:42.723951 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.723822    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14d0074c-a3ab-4ec3-b8e8-95b4681260fd" containerName="maas-api"
Oct 30 14:06:42.723951 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.723928    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="b819faba-b9f4-488f-bfa0-4da40fb59313" containerName="maas-api"
Oct 30 14:06:42.723951 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.723939    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="b676aeef-6a09-4e04-929d-6388b8034eac" containerName="maas-api"
Oct 30 14:06:42.723951 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.723946    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="70b59b98-9888-4d17-8695-e16a21f08fb0" containerName="payload-pre-processing"
Oct 30 14:06:42.723951 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.723953    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bcb5af7-5a52-48b1-88fa-e5b1359baced" containerName="payload-processing"
Oct 30 14:06:42.724534 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.723963    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="c04d4128-e9ec-4eeb-8eba-5d6d6c760ccd" containerName="maas-api"
Oct 30 14:06:42.724534 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.723976    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="63b21cf6-7b3b-48b6-88a1-d03e804d33fa" containerName="payload-pre-processing"
Oct 30 14:06:42.724534 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.723987    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="958c7031-f31a-4c34-9dd3-d01530719385" containerName="payload-processing"
Oct 30 14:06:42.724534 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.723994    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="00f2b03f-864c-4e03-9144-b949bced2ba9" containerName="maas-api"
Oct 30 14:06:42.724534 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.723999    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd554e69-a448-4d73-885a-d1ffcc724c56" containerName="payload-pre-processing"
Oct 30 14:06:42.724534 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.724007    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="14d0074c-a3ab-4ec3-b8e8-95b4681260fd" containerName="maas-api"
Oct 30 14:06:42.730228 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.730199    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-51dc9a/rate-test-model-51dc9a-kserve-69b7d876df-gqk6q"
Oct 30 14:06:42.748322 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.748289    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-51dc9a/rate-test-model-51dc9a-kserve-69b7d876df-gqk6q"]
Oct 30 14:06:42.760170 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.760120    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-51dc9a\"/\"openshift-service-ca.crt\""
Oct 30 14:06:42.760347 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.760173    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-51dc9a\"/\"kube-root-ca.crt\""
Oct 30 14:06:42.762675 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.762596    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-51dc9a\"/\"default-dockercfg-glfkt\""
Oct 30 14:06:42.762905 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.762885    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-51dc9a\"/\"rate-test-model-51dc9a-kserve-self-signed-certs\""
Oct 30 14:06:42.888714 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.888680    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16483221-0a88-4aa3-a47d-6d119206b423-tls-certs\") pod \"rate-test-model-51dc9a-kserve-69b7d876df-gqk6q\" (UID: \"16483221-0a88-4aa3-a47d-6d119206b423\") " pod="ai-tenant-e2e-shared-b-w0-51dc9a/rate-test-model-51dc9a-kserve-69b7d876df-gqk6q"
Oct 30 14:06:42.888903 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.888740    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16483221-0a88-4aa3-a47d-6d119206b423-tmp-dir\") pod \"rate-test-model-51dc9a-kserve-69b7d876df-gqk6q\" (UID: \"16483221-0a88-4aa3-a47d-6d119206b423\") " pod="ai-tenant-e2e-shared-b-w0-51dc9a/rate-test-model-51dc9a-kserve-69b7d876df-gqk6q"
Oct 30 14:06:42.888903 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.888872    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16483221-0a88-4aa3-a47d-6d119206b423-model-cache\") pod \"rate-test-model-51dc9a-kserve-69b7d876df-gqk6q\" (UID: \"16483221-0a88-4aa3-a47d-6d119206b423\") " pod="ai-tenant-e2e-shared-b-w0-51dc9a/rate-test-model-51dc9a-kserve-69b7d876df-gqk6q"
Oct 30 14:06:42.889025 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.888926    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16483221-0a88-4aa3-a47d-6d119206b423-home\") pod \"rate-test-model-51dc9a-kserve-69b7d876df-gqk6q\" (UID: \"16483221-0a88-4aa3-a47d-6d119206b423\") " pod="ai-tenant-e2e-shared-b-w0-51dc9a/rate-test-model-51dc9a-kserve-69b7d876df-gqk6q"
Oct 30 14:06:42.889025 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.888987    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5ckl\" (UniqueName: \"kubernetes.io/projected/16483221-0a88-4aa3-a47d-6d119206b423-kube-api-access-w5ckl\") pod \"rate-test-model-51dc9a-kserve-69b7d876df-gqk6q\" (UID: \"16483221-0a88-4aa3-a47d-6d119206b423\") " pod="ai-tenant-e2e-shared-b-w0-51dc9a/rate-test-model-51dc9a-kserve-69b7d876df-gqk6q"
Oct 30 14:06:42.889109 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.889076    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16483221-0a88-4aa3-a47d-6d119206b423-dshm\") pod \"rate-test-model-51dc9a-kserve-69b7d876df-gqk6q\" (UID: \"16483221-0a88-4aa3-a47d-6d119206b423\") " pod="ai-tenant-e2e-shared-b-w0-51dc9a/rate-test-model-51dc9a-kserve-69b7d876df-gqk6q"
Oct 30 14:06:42.990993 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.990875    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16483221-0a88-4aa3-a47d-6d119206b423-tls-certs\") pod \"rate-test-model-51dc9a-kserve-69b7d876df-gqk6q\" (UID: \"16483221-0a88-4aa3-a47d-6d119206b423\") " pod="ai-tenant-e2e-shared-b-w0-51dc9a/rate-test-model-51dc9a-kserve-69b7d876df-gqk6q"
Oct 30 14:06:42.990993 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.990960    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16483221-0a88-4aa3-a47d-6d119206b423-tmp-dir\") pod \"rate-test-model-51dc9a-kserve-69b7d876df-gqk6q\" (UID: \"16483221-0a88-4aa3-a47d-6d119206b423\") " pod="ai-tenant-e2e-shared-b-w0-51dc9a/rate-test-model-51dc9a-kserve-69b7d876df-gqk6q"
Oct 30 14:06:42.990993 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.990992    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16483221-0a88-4aa3-a47d-6d119206b423-model-cache\") pod \"rate-test-model-51dc9a-kserve-69b7d876df-gqk6q\" (UID: \"16483221-0a88-4aa3-a47d-6d119206b423\") " pod="ai-tenant-e2e-shared-b-w0-51dc9a/rate-test-model-51dc9a-kserve-69b7d876df-gqk6q"
Oct 30 14:06:42.991406 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.991029    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16483221-0a88-4aa3-a47d-6d119206b423-home\") pod \"rate-test-model-51dc9a-kserve-69b7d876df-gqk6q\" (UID: \"16483221-0a88-4aa3-a47d-6d119206b423\") " pod="ai-tenant-e2e-shared-b-w0-51dc9a/rate-test-model-51dc9a-kserve-69b7d876df-gqk6q"
Oct 30 14:06:42.991406 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.991100    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5ckl\" (UniqueName: \"kubernetes.io/projected/16483221-0a88-4aa3-a47d-6d119206b423-kube-api-access-w5ckl\") pod \"rate-test-model-51dc9a-kserve-69b7d876df-gqk6q\" (UID: \"16483221-0a88-4aa3-a47d-6d119206b423\") " pod="ai-tenant-e2e-shared-b-w0-51dc9a/rate-test-model-51dc9a-kserve-69b7d876df-gqk6q"
Oct 30 14:06:42.991406 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.991231    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16483221-0a88-4aa3-a47d-6d119206b423-dshm\") pod \"rate-test-model-51dc9a-kserve-69b7d876df-gqk6q\" (UID: \"16483221-0a88-4aa3-a47d-6d119206b423\") " pod="ai-tenant-e2e-shared-b-w0-51dc9a/rate-test-model-51dc9a-kserve-69b7d876df-gqk6q"
Oct 30 14:06:42.991944 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.991911    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16483221-0a88-4aa3-a47d-6d119206b423-model-cache\") pod \"rate-test-model-51dc9a-kserve-69b7d876df-gqk6q\" (UID: \"16483221-0a88-4aa3-a47d-6d119206b423\") " pod="ai-tenant-e2e-shared-b-w0-51dc9a/rate-test-model-51dc9a-kserve-69b7d876df-gqk6q"
Oct 30 14:06:42.992077 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.991991    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16483221-0a88-4aa3-a47d-6d119206b423-home\") pod \"rate-test-model-51dc9a-kserve-69b7d876df-gqk6q\" (UID: \"16483221-0a88-4aa3-a47d-6d119206b423\") " pod="ai-tenant-e2e-shared-b-w0-51dc9a/rate-test-model-51dc9a-kserve-69b7d876df-gqk6q"
Oct 30 14:06:42.992311 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.992282    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16483221-0a88-4aa3-a47d-6d119206b423-tmp-dir\") pod \"rate-test-model-51dc9a-kserve-69b7d876df-gqk6q\" (UID: \"16483221-0a88-4aa3-a47d-6d119206b423\") " pod="ai-tenant-e2e-shared-b-w0-51dc9a/rate-test-model-51dc9a-kserve-69b7d876df-gqk6q"
Oct 30 14:06:42.995304 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:42.995269    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16483221-0a88-4aa3-a47d-6d119206b423-dshm\") pod \"rate-test-model-51dc9a-kserve-69b7d876df-gqk6q\" (UID: \"16483221-0a88-4aa3-a47d-6d119206b423\") " pod="ai-tenant-e2e-shared-b-w0-51dc9a/rate-test-model-51dc9a-kserve-69b7d876df-gqk6q"
Oct 30 14:06:43.006075 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:43.006023    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00f2b03f-864c-4e03-9144-b949bced2ba9" path="/var/lib/kubelet/pods/00f2b03f-864c-4e03-9144-b949bced2ba9/volumes"
Oct 30 14:06:43.026543 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:43.025176    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16483221-0a88-4aa3-a47d-6d119206b423-tls-certs\") pod \"rate-test-model-51dc9a-kserve-69b7d876df-gqk6q\" (UID: \"16483221-0a88-4aa3-a47d-6d119206b423\") " pod="ai-tenant-e2e-shared-b-w0-51dc9a/rate-test-model-51dc9a-kserve-69b7d876df-gqk6q"
Oct 30 14:06:43.026543 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:43.025517    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5ckl\" (UniqueName: \"kubernetes.io/projected/16483221-0a88-4aa3-a47d-6d119206b423-kube-api-access-w5ckl\") pod \"rate-test-model-51dc9a-kserve-69b7d876df-gqk6q\" (UID: \"16483221-0a88-4aa3-a47d-6d119206b423\") " pod="ai-tenant-e2e-shared-b-w0-51dc9a/rate-test-model-51dc9a-kserve-69b7d876df-gqk6q"
Oct 30 14:06:43.052378 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:43.052326    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-51dc9a/rate-test-model-51dc9a-kserve-69b7d876df-gqk6q"
Oct 30 14:06:43.371318 ip-10-0-134-31 kubenswrapper[5093]: W1030 14:06:43.371277    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16483221_0a88_4aa3_a47d_6d119206b423.slice/crio-65f155680c4da839b8193e02eb304be5cfd20a091494a35ab690f79c5232f1b7 WatchSource:0}: Error finding container 65f155680c4da839b8193e02eb304be5cfd20a091494a35ab690f79c5232f1b7: Status 404 returned error can't find the container with id 65f155680c4da839b8193e02eb304be5cfd20a091494a35ab690f79c5232f1b7
Oct 30 14:06:43.373297 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:43.373145    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-51dc9a/rate-test-model-51dc9a-kserve-69b7d876df-gqk6q"]
Oct 30 14:06:44.074069 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:44.074024    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-51dc9a/rate-test-model-51dc9a-kserve-69b7d876df-gqk6q" event={"ID":"16483221-0a88-4aa3-a47d-6d119206b423","Type":"ContainerStarted","Data":"35252b57dac76b6bcfda78df08f1f4ea76f7db39fb47c4b3f72a88771d491504"}
Oct 30 14:06:44.074525 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:44.074077    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-51dc9a/rate-test-model-51dc9a-kserve-69b7d876df-gqk6q" event={"ID":"16483221-0a88-4aa3-a47d-6d119206b423","Type":"ContainerStarted","Data":"65f155680c4da839b8193e02eb304be5cfd20a091494a35ab690f79c5232f1b7"}
Oct 30 14:06:44.131820 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:44.131766    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-51dc9a/rate-test-model-51dc9a-kserve-69b7d876df-gqk6q" podStartSLOduration=2.131752699 podStartE2EDuration="2.131752699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:06:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:06:44.128065141 +0000 UTC m=+3823.275332380" watchObservedRunningTime="2026-10-30 14:06:44.131752699 +0000 UTC m=+3823.279019940"
Oct 30 14:06:44.484798 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:44.484771    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp"
Oct 30 14:06:44.614308 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:44.614258    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc75f0e6-64c6-4fde-9476-3859350da986-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc75f0e6-64c6-4fde-9476-3859350da986-istiod-ca-cert\") pod \"cc75f0e6-64c6-4fde-9476-3859350da986\" (UID: \"cc75f0e6-64c6-4fde-9476-3859350da986\") "
Oct 30 14:06:44.614475 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:44.614322    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc75f0e6-64c6-4fde-9476-3859350da986-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc75f0e6-64c6-4fde-9476-3859350da986-workload-certs\") pod \"cc75f0e6-64c6-4fde-9476-3859350da986\" (UID: \"cc75f0e6-64c6-4fde-9476-3859350da986\") "
Oct 30 14:06:44.614475 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:44.614350    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc75f0e6-64c6-4fde-9476-3859350da986-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc75f0e6-64c6-4fde-9476-3859350da986-istio-envoy\") pod \"cc75f0e6-64c6-4fde-9476-3859350da986\" (UID: \"cc75f0e6-64c6-4fde-9476-3859350da986\") "
Oct 30 14:06:44.614475 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:44.614453    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc75f0e6-64c6-4fde-9476-3859350da986-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc75f0e6-64c6-4fde-9476-3859350da986-workload-socket\") pod \"cc75f0e6-64c6-4fde-9476-3859350da986\" (UID: \"cc75f0e6-64c6-4fde-9476-3859350da986\") "
Oct 30 14:06:44.614604 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:44.614497    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc75f0e6-64c6-4fde-9476-3859350da986-kube-api-access-k2d6t\" (UniqueName: \"kubernetes.io/projected/cc75f0e6-64c6-4fde-9476-3859350da986-kube-api-access-k2d6t\") pod \"cc75f0e6-64c6-4fde-9476-3859350da986\" (UID: \"cc75f0e6-64c6-4fde-9476-3859350da986\") "
Oct 30 14:06:44.614646 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:44.614610    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc75f0e6-64c6-4fde-9476-3859350da986-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc75f0e6-64c6-4fde-9476-3859350da986-istio-podinfo\") pod \"cc75f0e6-64c6-4fde-9476-3859350da986\" (UID: \"cc75f0e6-64c6-4fde-9476-3859350da986\") "
Oct 30 14:06:44.614702 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:44.614675    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc75f0e6-64c6-4fde-9476-3859350da986-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc75f0e6-64c6-4fde-9476-3859350da986-credential-socket\") pod \"cc75f0e6-64c6-4fde-9476-3859350da986\" (UID: \"cc75f0e6-64c6-4fde-9476-3859350da986\") "
Oct 30 14:06:44.614702 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:44.614693    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc75f0e6-64c6-4fde-9476-3859350da986-workload-certs" pod "cc75f0e6-64c6-4fde-9476-3859350da986" (UID: "cc75f0e6-64c6-4fde-9476-3859350da986"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:06:44.614855 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:44.614718    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc75f0e6-64c6-4fde-9476-3859350da986-istiod-ca-cert" pod "cc75f0e6-64c6-4fde-9476-3859350da986" (UID: "cc75f0e6-64c6-4fde-9476-3859350da986"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:06:44.614855 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:44.614732    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc75f0e6-64c6-4fde-9476-3859350da986-workload-socket" pod "cc75f0e6-64c6-4fde-9476-3859350da986" (UID: "cc75f0e6-64c6-4fde-9476-3859350da986"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:06:44.614855 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:44.614779    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc75f0e6-64c6-4fde-9476-3859350da986-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc75f0e6-64c6-4fde-9476-3859350da986-istio-data\") pod \"cc75f0e6-64c6-4fde-9476-3859350da986\" (UID: \"cc75f0e6-64c6-4fde-9476-3859350da986\") "
Oct 30 14:06:44.614855 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:44.614825    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc75f0e6-64c6-4fde-9476-3859350da986-istio-token\" (UniqueName: \"kubernetes.io/projected/cc75f0e6-64c6-4fde-9476-3859350da986-istio-token\") pod \"cc75f0e6-64c6-4fde-9476-3859350da986\" (UID: \"cc75f0e6-64c6-4fde-9476-3859350da986\") "
Oct 30 14:06:44.615084 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:44.615024    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc75f0e6-64c6-4fde-9476-3859350da986-istio-data" pod "cc75f0e6-64c6-4fde-9476-3859350da986" (UID: "cc75f0e6-64c6-4fde-9476-3859350da986"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:06:44.615084 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:44.615041    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc75f0e6-64c6-4fde-9476-3859350da986-credential-socket" pod "cc75f0e6-64c6-4fde-9476-3859350da986" (UID: "cc75f0e6-64c6-4fde-9476-3859350da986"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:06:44.615491 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:44.615466    5093 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc75f0e6-64c6-4fde-9476-3859350da986-credential-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:06:44.615572 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:44.615494    5093 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc75f0e6-64c6-4fde-9476-3859350da986-istio-data\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:06:44.615572 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:44.615504    5093 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc75f0e6-64c6-4fde-9476-3859350da986-istiod-ca-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:06:44.615572 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:44.615512    5093 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc75f0e6-64c6-4fde-9476-3859350da986-workload-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:06:44.615572 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:44.615521    5093 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc75f0e6-64c6-4fde-9476-3859350da986-workload-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:06:44.617224 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:44.617202    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc75f0e6-64c6-4fde-9476-3859350da986-istio-podinfo" pod "cc75f0e6-64c6-4fde-9476-3859350da986" (UID: "cc75f0e6-64c6-4fde-9476-3859350da986"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:06:44.617224 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:44.617211    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc75f0e6-64c6-4fde-9476-3859350da986-istio-envoy" pod "cc75f0e6-64c6-4fde-9476-3859350da986" (UID: "cc75f0e6-64c6-4fde-9476-3859350da986"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:06:44.617414 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:44.617396    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc75f0e6-64c6-4fde-9476-3859350da986-kube-api-access-k2d6t" pod "cc75f0e6-64c6-4fde-9476-3859350da986" (UID: "cc75f0e6-64c6-4fde-9476-3859350da986"). InnerVolumeSpecName "kube-api-access-k2d6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:06:44.617477 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:44.617435    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc75f0e6-64c6-4fde-9476-3859350da986-istio-token" pod "cc75f0e6-64c6-4fde-9476-3859350da986" (UID: "cc75f0e6-64c6-4fde-9476-3859350da986"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:06:44.716571 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:44.716534    5093 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc75f0e6-64c6-4fde-9476-3859350da986-istio-podinfo\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:06:44.716571 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:44.716565    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc75f0e6-64c6-4fde-9476-3859350da986-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:06:44.716571 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:44.716576    5093 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc75f0e6-64c6-4fde-9476-3859350da986-istio-envoy\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:06:44.716835 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:44.716584    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2d6t\" (UniqueName: \"kubernetes.io/projected/cc75f0e6-64c6-4fde-9476-3859350da986-kube-api-access-k2d6t\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:06:45.053188 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:45.053146    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-51dc9a/rate-test-model-51dc9a-kserve-69b7d876df-gqk6q"
Oct 30 14:06:45.082678 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:45.082641    5093 generic.go:356] "Generic (PLEG): container finished" podID="cc75f0e6-64c6-4fde-9476-3859350da986" containerID="9826755f30199716a4db91c54e8186873daa229ff6f3eb83d664150033f7454f" exitCode=0
Oct 30 14:06:45.083197 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:45.082717    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp"
Oct 30 14:06:45.083197 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:45.082713    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp" event={"ID":"cc75f0e6-64c6-4fde-9476-3859350da986","Type":"ContainerDied","Data":"9826755f30199716a4db91c54e8186873daa229ff6f3eb83d664150033f7454f"}
Oct 30 14:06:45.083197 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:45.082869    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp" event={"ID":"cc75f0e6-64c6-4fde-9476-3859350da986","Type":"ContainerDied","Data":"0bfd8d4b3904165b722b499849aada63d32fe050f73b08dca945ab022560cbc5"}
Oct 30 14:06:45.083197 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:45.082903    5093 scope.go:122] "RemoveContainer" containerID="9826755f30199716a4db91c54e8186873daa229ff6f3eb83d664150033f7454f"
Oct 30 14:06:45.095113 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:45.095092    5093 scope.go:122] "RemoveContainer" containerID="b2c240320d8110c9d359ce33e13d8f8d1fde4db9e02e3c491dab4ca21fff450b"
Oct 30 14:06:45.108081 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:45.108053    5093 scope.go:122] "RemoveContainer" containerID="9826755f30199716a4db91c54e8186873daa229ff6f3eb83d664150033f7454f"
Oct 30 14:06:45.108666 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:06:45.108640    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9826755f30199716a4db91c54e8186873daa229ff6f3eb83d664150033f7454f\": container with ID starting with 9826755f30199716a4db91c54e8186873daa229ff6f3eb83d664150033f7454f not found: ID does not exist" containerID="9826755f30199716a4db91c54e8186873daa229ff6f3eb83d664150033f7454f"
Oct 30 14:06:45.108773 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:45.108673    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9826755f30199716a4db91c54e8186873daa229ff6f3eb83d664150033f7454f"} err="failed to get container status \"9826755f30199716a4db91c54e8186873daa229ff6f3eb83d664150033f7454f\": rpc error: code = NotFound desc = could not find container \"9826755f30199716a4db91c54e8186873daa229ff6f3eb83d664150033f7454f\": container with ID starting with 9826755f30199716a4db91c54e8186873daa229ff6f3eb83d664150033f7454f not found: ID does not exist"
Oct 30 14:06:45.108773 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:45.108691    5093 scope.go:122] "RemoveContainer" containerID="b2c240320d8110c9d359ce33e13d8f8d1fde4db9e02e3c491dab4ca21fff450b"
Oct 30 14:06:45.109003 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:06:45.108981    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2c240320d8110c9d359ce33e13d8f8d1fde4db9e02e3c491dab4ca21fff450b\": container with ID starting with b2c240320d8110c9d359ce33e13d8f8d1fde4db9e02e3c491dab4ca21fff450b not found: ID does not exist" containerID="b2c240320d8110c9d359ce33e13d8f8d1fde4db9e02e3c491dab4ca21fff450b"
Oct 30 14:06:45.109047 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:45.109010    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2c240320d8110c9d359ce33e13d8f8d1fde4db9e02e3c491dab4ca21fff450b"} err="failed to get container status \"b2c240320d8110c9d359ce33e13d8f8d1fde4db9e02e3c491dab4ca21fff450b\": rpc error: code = NotFound desc = could not find container \"b2c240320d8110c9d359ce33e13d8f8d1fde4db9e02e3c491dab4ca21fff450b\": container with ID starting with b2c240320d8110c9d359ce33e13d8f8d1fde4db9e02e3c491dab4ca21fff450b not found: ID does not exist"
Oct 30 14:06:45.155548 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:45.155510    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp"]
Oct 30 14:06:45.167550 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:45.167514    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-dgsgp"]
Oct 30 14:06:47.018010 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:47.016095    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc75f0e6-64c6-4fde-9476-3859350da986" path="/var/lib/kubelet/pods/cc75f0e6-64c6-4fde-9476-3859350da986/volumes"
Oct 30 14:06:47.923020 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:47.922970    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-bd2fed-68ff7bd4b6-d9tt2" podUID="64b19bcc-57c8-4838-a94c-5b918d4da2e2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:06:49.389794 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:49.389747    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9b32c32a-75dd97f79d-t6tgs"]
Oct 30 14:06:49.423944 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:49.423909    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9b32c32a-6d586d78d7-qvxvx"]
Oct 30 14:06:50.296747 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:50.296711    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp"]
Oct 30 14:06:50.297006 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:50.296971    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp" podUID="18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e" containerName="istio-proxy" containerID="cri-o://3ff07d3b7abb82a8b1e62ed1fadc16ab071c289b98414c744a1ad42ea3357287" gracePeriod=30
Oct 30 14:06:51.170218 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:06:51.170176    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 14:06:53.052930 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:53.052892    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-51dc9a/rate-test-model-51dc9a-kserve-69b7d876df-gqk6q"
Oct 30 14:06:53.066033 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:53.066002    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-51dc9a/rate-test-model-51dc9a-kserve-69b7d876df-gqk6q"
Oct 30 14:06:53.160377 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:53.160341    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-51dc9a/rate-test-model-51dc9a-kserve-69b7d876df-gqk6q"
Oct 30 14:06:54.006847 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:54.006819    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-bd2fed-68ff7bd4b6-d9tt2"
Oct 30 14:06:54.058020 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:54.057985    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64b19bcc-57c8-4838-a94c-5b918d4da2e2-kube-api-access-4b9r7\" (UniqueName: \"kubernetes.io/projected/64b19bcc-57c8-4838-a94c-5b918d4da2e2-kube-api-access-4b9r7\") pod \"64b19bcc-57c8-4838-a94c-5b918d4da2e2\" (UID: \"64b19bcc-57c8-4838-a94c-5b918d4da2e2\") "
Oct 30 14:06:54.058725 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:54.058072    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64b19bcc-57c8-4838-a94c-5b918d4da2e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64b19bcc-57c8-4838-a94c-5b918d4da2e2-plugins-config-volume\") pod \"64b19bcc-57c8-4838-a94c-5b918d4da2e2\" (UID: \"64b19bcc-57c8-4838-a94c-5b918d4da2e2\") "
Oct 30 14:06:54.058811 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:54.058735    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64b19bcc-57c8-4838-a94c-5b918d4da2e2-plugins-config-volume" pod "64b19bcc-57c8-4838-a94c-5b918d4da2e2" (UID: "64b19bcc-57c8-4838-a94c-5b918d4da2e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:06:54.061414 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:54.061385    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64b19bcc-57c8-4838-a94c-5b918d4da2e2-kube-api-access-4b9r7" pod "64b19bcc-57c8-4838-a94c-5b918d4da2e2" (UID: "64b19bcc-57c8-4838-a94c-5b918d4da2e2"). InnerVolumeSpecName "kube-api-access-4b9r7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:06:54.159931 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:54.159886    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b9r7\" (UniqueName: \"kubernetes.io/projected/64b19bcc-57c8-4838-a94c-5b918d4da2e2-kube-api-access-4b9r7\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:06:54.159931 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:54.159921    5093 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64b19bcc-57c8-4838-a94c-5b918d4da2e2-plugins-config-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:06:54.160568 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:54.160530    5093 generic.go:356] "Generic (PLEG): container finished" podID="64b19bcc-57c8-4838-a94c-5b918d4da2e2" containerID="8a527cd176c5ede8c33363022a30d2134b4b009b09d12b4fa5e36779313431da" exitCode=0
Oct 30 14:06:54.160729 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:54.160590    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-bd2fed-68ff7bd4b6-d9tt2"
Oct 30 14:06:54.160729 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:54.160616    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-bd2fed-68ff7bd4b6-d9tt2" event={"ID":"64b19bcc-57c8-4838-a94c-5b918d4da2e2","Type":"ContainerDied","Data":"8a527cd176c5ede8c33363022a30d2134b4b009b09d12b4fa5e36779313431da"}
Oct 30 14:06:54.160729 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:54.160684    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-bd2fed-68ff7bd4b6-d9tt2" event={"ID":"64b19bcc-57c8-4838-a94c-5b918d4da2e2","Type":"ContainerDied","Data":"99d8b332d357ab6a771d5d5835814116e9ad9b4571cd4259fe9a73b22eb2a329"}
Oct 30 14:06:54.160729 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:54.160713    5093 scope.go:122] "RemoveContainer" containerID="8a527cd176c5ede8c33363022a30d2134b4b009b09d12b4fa5e36779313431da"
Oct 30 14:06:54.174154 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:54.174118    5093 scope.go:122] "RemoveContainer" containerID="8a527cd176c5ede8c33363022a30d2134b4b009b09d12b4fa5e36779313431da"
Oct 30 14:06:54.174455 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:06:54.174435    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a527cd176c5ede8c33363022a30d2134b4b009b09d12b4fa5e36779313431da\": container with ID starting with 8a527cd176c5ede8c33363022a30d2134b4b009b09d12b4fa5e36779313431da not found: ID does not exist" containerID="8a527cd176c5ede8c33363022a30d2134b4b009b09d12b4fa5e36779313431da"
Oct 30 14:06:54.174518 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:54.174463    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a527cd176c5ede8c33363022a30d2134b4b009b09d12b4fa5e36779313431da"} err="failed to get container status \"8a527cd176c5ede8c33363022a30d2134b4b009b09d12b4fa5e36779313431da\": rpc error: code = NotFound desc = could not find container \"8a527cd176c5ede8c33363022a30d2134b4b009b09d12b4fa5e36779313431da\": container with ID starting with 8a527cd176c5ede8c33363022a30d2134b4b009b09d12b4fa5e36779313431da not found: ID does not exist"
Oct 30 14:06:54.217638 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:54.217601    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-bd2fed-68ff7bd4b6-d9tt2"]
Oct 30 14:06:54.222711 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:54.222644    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-bd2fed-68ff7bd4b6-d9tt2"]
Oct 30 14:06:54.390903 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:54.390858    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-9b32c32a-75dd97f79d-t6tgs" podUID="c822e501-9472-4c26-9be9-c308c69fccbb" containerName="payload-processing" containerID="cri-o://39a40002639e7e2c2148694586077e789d4b5cbda6308ac09e87341e28b93c5e" gracePeriod=25
Oct 30 14:06:54.424262 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:54.424226    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-9b32c32a-6d586d78d7-qvxvx" podUID="f4e930a4-18f6-49b6-938d-b2d4243643c7" containerName="payload-pre-processing" containerID="cri-o://4e833ad80ec132d15d5b41fad3b633da5ea442a4c6bf1c951c6c0280ce4c48b9" gracePeriod=25
Oct 30 14:06:54.716187 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:54.716160    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9b32c32a-75dd97f79d-t6tgs"
Oct 30 14:06:54.723484 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:54.723458    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9b32c32a-6d586d78d7-qvxvx"
Oct 30 14:06:54.769576 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:54.769539    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4e930a4-18f6-49b6-938d-b2d4243643c7-kube-api-access-fcxl5\" (UniqueName: \"kubernetes.io/projected/f4e930a4-18f6-49b6-938d-b2d4243643c7-kube-api-access-fcxl5\") pod \"f4e930a4-18f6-49b6-938d-b2d4243643c7\" (UID: \"f4e930a4-18f6-49b6-938d-b2d4243643c7\") "
Oct 30 14:06:54.769765 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:54.769612    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c822e501-9472-4c26-9be9-c308c69fccbb-kube-api-access-s6qg6\" (UniqueName: \"kubernetes.io/projected/c822e501-9472-4c26-9be9-c308c69fccbb-kube-api-access-s6qg6\") pod \"c822e501-9472-4c26-9be9-c308c69fccbb\" (UID: \"c822e501-9472-4c26-9be9-c308c69fccbb\") "
Oct 30 14:06:54.769765 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:54.769669    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4e930a4-18f6-49b6-938d-b2d4243643c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4e930a4-18f6-49b6-938d-b2d4243643c7-plugins-config-volume\") pod \"f4e930a4-18f6-49b6-938d-b2d4243643c7\" (UID: \"f4e930a4-18f6-49b6-938d-b2d4243643c7\") "
Oct 30 14:06:54.769903 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:54.769791    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c822e501-9472-4c26-9be9-c308c69fccbb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c822e501-9472-4c26-9be9-c308c69fccbb-plugins-config-volume\") pod \"c822e501-9472-4c26-9be9-c308c69fccbb\" (UID: \"c822e501-9472-4c26-9be9-c308c69fccbb\") "
Oct 30 14:06:54.770489 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:54.770457    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4e930a4-18f6-49b6-938d-b2d4243643c7-plugins-config-volume" pod "f4e930a4-18f6-49b6-938d-b2d4243643c7" (UID: "f4e930a4-18f6-49b6-938d-b2d4243643c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:06:54.770702 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:54.770680    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c822e501-9472-4c26-9be9-c308c69fccbb-plugins-config-volume" pod "c822e501-9472-4c26-9be9-c308c69fccbb" (UID: "c822e501-9472-4c26-9be9-c308c69fccbb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:06:54.772796 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:54.772766    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c822e501-9472-4c26-9be9-c308c69fccbb-kube-api-access-s6qg6" pod "c822e501-9472-4c26-9be9-c308c69fccbb" (UID: "c822e501-9472-4c26-9be9-c308c69fccbb"). InnerVolumeSpecName "kube-api-access-s6qg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:06:54.772902 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:54.772824    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4e930a4-18f6-49b6-938d-b2d4243643c7-kube-api-access-fcxl5" pod "f4e930a4-18f6-49b6-938d-b2d4243643c7" (UID: "f4e930a4-18f6-49b6-938d-b2d4243643c7"). InnerVolumeSpecName "kube-api-access-fcxl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:06:54.874153 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:54.873665    5093 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4e930a4-18f6-49b6-938d-b2d4243643c7-plugins-config-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:06:54.874153 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:54.873779    5093 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c822e501-9472-4c26-9be9-c308c69fccbb-plugins-config-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:06:54.874153 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:54.873796    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcxl5\" (UniqueName: \"kubernetes.io/projected/f4e930a4-18f6-49b6-938d-b2d4243643c7-kube-api-access-fcxl5\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:06:54.874153 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:54.873822    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6qg6\" (UniqueName: \"kubernetes.io/projected/c822e501-9472-4c26-9be9-c308c69fccbb-kube-api-access-s6qg6\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:06:55.000604 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:55.000565    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64b19bcc-57c8-4838-a94c-5b918d4da2e2" path="/var/lib/kubelet/pods/64b19bcc-57c8-4838-a94c-5b918d4da2e2/volumes"
Oct 30 14:06:55.170418 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:55.170313    5093 generic.go:356] "Generic (PLEG): container finished" podID="f4e930a4-18f6-49b6-938d-b2d4243643c7" containerID="4e833ad80ec132d15d5b41fad3b633da5ea442a4c6bf1c951c6c0280ce4c48b9" exitCode=0
Oct 30 14:06:55.170418 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:55.170364    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9b32c32a-6d586d78d7-qvxvx"
Oct 30 14:06:55.171015 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:55.170403    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9b32c32a-6d586d78d7-qvxvx" event={"ID":"f4e930a4-18f6-49b6-938d-b2d4243643c7","Type":"ContainerDied","Data":"4e833ad80ec132d15d5b41fad3b633da5ea442a4c6bf1c951c6c0280ce4c48b9"}
Oct 30 14:06:55.171015 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:55.170477    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9b32c32a-6d586d78d7-qvxvx" event={"ID":"f4e930a4-18f6-49b6-938d-b2d4243643c7","Type":"ContainerDied","Data":"7a9552092ddf4c77726d099a98b3f7c2e8f3eb5c5673e74dfc846795d2047058"}
Oct 30 14:06:55.171015 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:55.170508    5093 scope.go:122] "RemoveContainer" containerID="4e833ad80ec132d15d5b41fad3b633da5ea442a4c6bf1c951c6c0280ce4c48b9"
Oct 30 14:06:55.174413 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:55.174390    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9b32c32a-75dd97f79d-t6tgs"
Oct 30 14:06:55.174989 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:55.174959    5093 generic.go:356] "Generic (PLEG): container finished" podID="c822e501-9472-4c26-9be9-c308c69fccbb" containerID="39a40002639e7e2c2148694586077e789d4b5cbda6308ac09e87341e28b93c5e" exitCode=0
Oct 30 14:06:55.175252 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:55.175056    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9b32c32a-75dd97f79d-t6tgs" event={"ID":"c822e501-9472-4c26-9be9-c308c69fccbb","Type":"ContainerDied","Data":"39a40002639e7e2c2148694586077e789d4b5cbda6308ac09e87341e28b93c5e"}
Oct 30 14:06:55.175252 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:55.175117    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9b32c32a-75dd97f79d-t6tgs" event={"ID":"c822e501-9472-4c26-9be9-c308c69fccbb","Type":"ContainerDied","Data":"f7fa68d4ad2342aa5266013b49a15a73cd320952c94c14227e2661dfc37976b3"}
Oct 30 14:06:55.183725 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:55.183704    5093 scope.go:122] "RemoveContainer" containerID="4e833ad80ec132d15d5b41fad3b633da5ea442a4c6bf1c951c6c0280ce4c48b9"
Oct 30 14:06:55.184078 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:06:55.184032    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e833ad80ec132d15d5b41fad3b633da5ea442a4c6bf1c951c6c0280ce4c48b9\": container with ID starting with 4e833ad80ec132d15d5b41fad3b633da5ea442a4c6bf1c951c6c0280ce4c48b9 not found: ID does not exist" containerID="4e833ad80ec132d15d5b41fad3b633da5ea442a4c6bf1c951c6c0280ce4c48b9"
Oct 30 14:06:55.184078 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:55.184068    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e833ad80ec132d15d5b41fad3b633da5ea442a4c6bf1c951c6c0280ce4c48b9"} err="failed to get container status \"4e833ad80ec132d15d5b41fad3b633da5ea442a4c6bf1c951c6c0280ce4c48b9\": rpc error: code = NotFound desc = could not find container \"4e833ad80ec132d15d5b41fad3b633da5ea442a4c6bf1c951c6c0280ce4c48b9\": container with ID starting with 4e833ad80ec132d15d5b41fad3b633da5ea442a4c6bf1c951c6c0280ce4c48b9 not found: ID does not exist"
Oct 30 14:06:55.184225 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:55.184085    5093 scope.go:122] "RemoveContainer" containerID="39a40002639e7e2c2148694586077e789d4b5cbda6308ac09e87341e28b93c5e"
Oct 30 14:06:55.194303 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:55.194283    5093 scope.go:122] "RemoveContainer" containerID="39a40002639e7e2c2148694586077e789d4b5cbda6308ac09e87341e28b93c5e"
Oct 30 14:06:55.194593 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:06:55.194575    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39a40002639e7e2c2148694586077e789d4b5cbda6308ac09e87341e28b93c5e\": container with ID starting with 39a40002639e7e2c2148694586077e789d4b5cbda6308ac09e87341e28b93c5e not found: ID does not exist" containerID="39a40002639e7e2c2148694586077e789d4b5cbda6308ac09e87341e28b93c5e"
Oct 30 14:06:55.194643 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:55.194601    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39a40002639e7e2c2148694586077e789d4b5cbda6308ac09e87341e28b93c5e"} err="failed to get container status \"39a40002639e7e2c2148694586077e789d4b5cbda6308ac09e87341e28b93c5e\": rpc error: code = NotFound desc = could not find container \"39a40002639e7e2c2148694586077e789d4b5cbda6308ac09e87341e28b93c5e\": container with ID starting with 39a40002639e7e2c2148694586077e789d4b5cbda6308ac09e87341e28b93c5e not found: ID does not exist"
Oct 30 14:06:55.225909 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:55.225871    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9b32c32a-75dd97f79d-t6tgs"]
Oct 30 14:06:55.241180 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:55.241142    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9b32c32a-75dd97f79d-t6tgs"]
Oct 30 14:06:55.269926 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:55.269699    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9b32c32a-6d586d78d7-qvxvx"]
Oct 30 14:06:55.273323 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:55.273291    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9b32c32a-6d586d78d7-qvxvx"]
Oct 30 14:06:55.583612 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:55.583583    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp"
Oct 30 14:06:55.694962 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:55.694918    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-istio-envoy\") pod \"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e\" (UID: \"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e\") "
Oct 30 14:06:55.695165 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:55.695063    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-istio-data\") pod \"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e\" (UID: \"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e\") "
Oct 30 14:06:55.695165 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:55.695121    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-kube-api-access-xp4k2\" (UniqueName: \"kubernetes.io/projected/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-kube-api-access-xp4k2\") pod \"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e\" (UID: \"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e\") "
Oct 30 14:06:55.695302 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:55.695197    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-credential-socket\") pod \"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e\" (UID: \"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e\") "
Oct 30 14:06:55.695302 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:55.695232    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-workload-socket\") pod \"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e\" (UID: \"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e\") "
Oct 30 14:06:55.695302 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:55.695285    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-workload-certs\") pod \"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e\" (UID: \"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e\") "
Oct 30 14:06:55.695443 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:55.695319    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-istiod-ca-cert\") pod \"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e\" (UID: \"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e\") "
Oct 30 14:06:55.695443 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:55.695348    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-istio-data" pod "18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e" (UID: "18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:06:55.695443 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:55.695420    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-istio-podinfo\") pod \"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e\" (UID: \"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e\") "
Oct 30 14:06:55.695602 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:55.695513    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-istio-token\" (UniqueName: \"kubernetes.io/projected/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-istio-token\") pod \"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e\" (UID: \"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e\") "
Oct 30 14:06:55.695667 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:55.695601    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-workload-socket" pod "18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e" (UID: "18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:06:55.695667 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:55.695617    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-workload-certs" pod "18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e" (UID: "18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:06:55.695888 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:55.695865    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-istiod-ca-cert" pod "18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e" (UID: "18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:06:55.696119 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:55.696099    5093 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-istio-data\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:06:55.696204 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:55.696121    5093 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-workload-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:06:55.696204 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:55.696149    5093 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-workload-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:06:55.696204 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:55.696163    5093 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-istiod-ca-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:06:55.696350 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:55.696332    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-credential-socket" pod "18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e" (UID: "18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:06:55.697633 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:55.697604    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-istio-envoy" pod "18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e" (UID: "18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:06:55.698246 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:55.698213    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-istio-token" pod "18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e" (UID: "18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:06:55.698351 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:55.698260    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-istio-podinfo" pod "18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e" (UID: "18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:06:55.698351 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:55.698321    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-kube-api-access-xp4k2" pod "18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e" (UID: "18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e"). InnerVolumeSpecName "kube-api-access-xp4k2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:06:55.796971 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:55.796929    5093 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-istio-podinfo\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:06:55.796971 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:55.796965    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:06:55.796971 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:55.796977    5093 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-istio-envoy\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:06:55.797273 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:55.796986    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp4k2\" (UniqueName: \"kubernetes.io/projected/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-kube-api-access-xp4k2\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:06:55.797273 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:55.796998    5093 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e-credential-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:06:56.186918 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:56.186878    5093 generic.go:356] "Generic (PLEG): container finished" podID="18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e" containerID="3ff07d3b7abb82a8b1e62ed1fadc16ab071c289b98414c744a1ad42ea3357287" exitCode=0
Oct 30 14:06:56.187368 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:56.186934    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp"
Oct 30 14:06:56.187368 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:56.186955    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp" event={"ID":"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e","Type":"ContainerDied","Data":"3ff07d3b7abb82a8b1e62ed1fadc16ab071c289b98414c744a1ad42ea3357287"}
Oct 30 14:06:56.187368 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:56.187020    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp" event={"ID":"18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e","Type":"ContainerDied","Data":"54ace99a523e8155f8f4a62fd88f48f4640e73defc78e0e8f0719c96532e287a"}
Oct 30 14:06:56.187368 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:56.187051    5093 scope.go:122] "RemoveContainer" containerID="3ff07d3b7abb82a8b1e62ed1fadc16ab071c289b98414c744a1ad42ea3357287"
Oct 30 14:06:56.198258 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:56.198233    5093 scope.go:122] "RemoveContainer" containerID="bf678e91aa9af56a31ab09de6b7287b69bc50de05e9cf7030530a2bf694faf47"
Oct 30 14:06:56.209541 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:56.209518    5093 scope.go:122] "RemoveContainer" containerID="3ff07d3b7abb82a8b1e62ed1fadc16ab071c289b98414c744a1ad42ea3357287"
Oct 30 14:06:56.209817 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:06:56.209796    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ff07d3b7abb82a8b1e62ed1fadc16ab071c289b98414c744a1ad42ea3357287\": container with ID starting with 3ff07d3b7abb82a8b1e62ed1fadc16ab071c289b98414c744a1ad42ea3357287 not found: ID does not exist" containerID="3ff07d3b7abb82a8b1e62ed1fadc16ab071c289b98414c744a1ad42ea3357287"
Oct 30 14:06:56.209893 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:56.209828    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ff07d3b7abb82a8b1e62ed1fadc16ab071c289b98414c744a1ad42ea3357287"} err="failed to get container status \"3ff07d3b7abb82a8b1e62ed1fadc16ab071c289b98414c744a1ad42ea3357287\": rpc error: code = NotFound desc = could not find container \"3ff07d3b7abb82a8b1e62ed1fadc16ab071c289b98414c744a1ad42ea3357287\": container with ID starting with 3ff07d3b7abb82a8b1e62ed1fadc16ab071c289b98414c744a1ad42ea3357287 not found: ID does not exist"
Oct 30 14:06:56.209893 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:56.209848    5093 scope.go:122] "RemoveContainer" containerID="bf678e91aa9af56a31ab09de6b7287b69bc50de05e9cf7030530a2bf694faf47"
Oct 30 14:06:56.210124 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:06:56.210098    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf678e91aa9af56a31ab09de6b7287b69bc50de05e9cf7030530a2bf694faf47\": container with ID starting with bf678e91aa9af56a31ab09de6b7287b69bc50de05e9cf7030530a2bf694faf47 not found: ID does not exist" containerID="bf678e91aa9af56a31ab09de6b7287b69bc50de05e9cf7030530a2bf694faf47"
Oct 30 14:06:56.210232 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:56.210152    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf678e91aa9af56a31ab09de6b7287b69bc50de05e9cf7030530a2bf694faf47"} err="failed to get container status \"bf678e91aa9af56a31ab09de6b7287b69bc50de05e9cf7030530a2bf694faf47\": rpc error: code = NotFound desc = could not find container \"bf678e91aa9af56a31ab09de6b7287b69bc50de05e9cf7030530a2bf694faf47\": container with ID starting with bf678e91aa9af56a31ab09de6b7287b69bc50de05e9cf7030530a2bf694faf47 not found: ID does not exist"
Oct 30 14:06:56.242099 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:56.242066    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp"]
Oct 30 14:06:56.256842 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:56.256808    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-bd2fed-openshift-default-66bd69dccc-rqdjp"]
Oct 30 14:06:56.999938 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:56.999904    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e" path="/var/lib/kubelet/pods/18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e/volumes"
Oct 30 14:06:57.000398 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:57.000383    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c822e501-9472-4c26-9be9-c308c69fccbb" path="/var/lib/kubelet/pods/c822e501-9472-4c26-9be9-c308c69fccbb/volumes"
Oct 30 14:06:57.000760 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:06:57.000748    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4e930a4-18f6-49b6-938d-b2d4243643c7" path="/var/lib/kubelet/pods/f4e930a4-18f6-49b6-938d-b2d4243643c7/volumes"
Oct 30 14:07:19.002973 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:19.002936    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-f7ee8a/rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn"]
Oct 30 14:07:19.003868 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:19.003827    5093 cpu_manager.go:414] "containerMap: removing container" podUID="18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e" containerName="istio-proxy"
Oct 30 14:07:19.003868 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:19.003852    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e" containerName="istio-proxy"
Oct 30 14:07:19.003868 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:19.003867    5093 cpu_manager.go:414] "containerMap: removing container" podUID="64b19bcc-57c8-4838-a94c-5b918d4da2e2" containerName="payload-processing"
Oct 30 14:07:19.003868 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:19.003874    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64b19bcc-57c8-4838-a94c-5b918d4da2e2" containerName="payload-processing"
Oct 30 14:07:19.004209 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:19.003892    5093 cpu_manager.go:414] "containerMap: removing container" podUID="cc75f0e6-64c6-4fde-9476-3859350da986" containerName="istio-proxy"
Oct 30 14:07:19.004209 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:19.003899    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc75f0e6-64c6-4fde-9476-3859350da986" containerName="istio-proxy"
Oct 30 14:07:19.004209 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:19.003928    5093 cpu_manager.go:414] "containerMap: removing container" podUID="f4e930a4-18f6-49b6-938d-b2d4243643c7" containerName="payload-pre-processing"
Oct 30 14:07:19.004209 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:19.003933    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4e930a4-18f6-49b6-938d-b2d4243643c7" containerName="payload-pre-processing"
Oct 30 14:07:19.004209 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:19.003952    5093 cpu_manager.go:414] "containerMap: removing container" podUID="c822e501-9472-4c26-9be9-c308c69fccbb" containerName="payload-processing"
Oct 30 14:07:19.004209 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:19.003957    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c822e501-9472-4c26-9be9-c308c69fccbb" containerName="payload-processing"
Oct 30 14:07:19.004209 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:19.004066    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e" containerName="istio-proxy"
Oct 30 14:07:19.004209 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:19.004075    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="64b19bcc-57c8-4838-a94c-5b918d4da2e2" containerName="payload-processing"
Oct 30 14:07:19.004209 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:19.004084    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc75f0e6-64c6-4fde-9476-3859350da986" containerName="istio-proxy"
Oct 30 14:07:19.004209 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:19.004094    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4e930a4-18f6-49b6-938d-b2d4243643c7" containerName="payload-pre-processing"
Oct 30 14:07:19.004209 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:19.004103    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="c822e501-9472-4c26-9be9-c308c69fccbb" containerName="payload-processing"
Oct 30 14:07:19.008189 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:19.008164    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-f7ee8a/rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn"
Oct 30 14:07:19.030003 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:19.029959    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-f7ee8a/rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn"]
Oct 30 14:07:19.038255 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:19.037191    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-f7ee8a\"/\"default-dockercfg-v895p\""
Oct 30 14:07:19.038255 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:19.037302    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-f7ee8a\"/\"kube-root-ca.crt\""
Oct 30 14:07:19.038255 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:19.037364    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-f7ee8a\"/\"openshift-service-ca.crt\""
Oct 30 14:07:19.038255 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:19.037457    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-f7ee8a\"/\"rate-test-model-f7ee8a-kserve-self-signed-certs\""
Oct 30 14:07:19.124456 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:19.124404    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/991b571e-8f93-47b0-9bb8-ab71ee26897c-tls-certs\") pod \"rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn\" (UID: \"991b571e-8f93-47b0-9bb8-ab71ee26897c\") " pod="ai-tenant-e2e-shared-a-w1-f7ee8a/rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn"
Oct 30 14:07:19.124657 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:19.124545    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgpws\" (UniqueName: \"kubernetes.io/projected/991b571e-8f93-47b0-9bb8-ab71ee26897c-kube-api-access-vgpws\") pod \"rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn\" (UID: \"991b571e-8f93-47b0-9bb8-ab71ee26897c\") " pod="ai-tenant-e2e-shared-a-w1-f7ee8a/rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn"
Oct 30 14:07:19.124725 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:19.124701    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/991b571e-8f93-47b0-9bb8-ab71ee26897c-home\") pod \"rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn\" (UID: \"991b571e-8f93-47b0-9bb8-ab71ee26897c\") " pod="ai-tenant-e2e-shared-a-w1-f7ee8a/rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn"
Oct 30 14:07:19.124773 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:19.124743    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/991b571e-8f93-47b0-9bb8-ab71ee26897c-model-cache\") pod \"rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn\" (UID: \"991b571e-8f93-47b0-9bb8-ab71ee26897c\") " pod="ai-tenant-e2e-shared-a-w1-f7ee8a/rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn"
Oct 30 14:07:19.124773 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:19.124760    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/991b571e-8f93-47b0-9bb8-ab71ee26897c-dshm\") pod \"rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn\" (UID: \"991b571e-8f93-47b0-9bb8-ab71ee26897c\") " pod="ai-tenant-e2e-shared-a-w1-f7ee8a/rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn"
Oct 30 14:07:19.124905 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:19.124886    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/991b571e-8f93-47b0-9bb8-ab71ee26897c-tmp-dir\") pod \"rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn\" (UID: \"991b571e-8f93-47b0-9bb8-ab71ee26897c\") " pod="ai-tenant-e2e-shared-a-w1-f7ee8a/rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn"
Oct 30 14:07:19.226546 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:19.226508    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgpws\" (UniqueName: \"kubernetes.io/projected/991b571e-8f93-47b0-9bb8-ab71ee26897c-kube-api-access-vgpws\") pod \"rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn\" (UID: \"991b571e-8f93-47b0-9bb8-ab71ee26897c\") " pod="ai-tenant-e2e-shared-a-w1-f7ee8a/rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn"
Oct 30 14:07:19.226757 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:19.226689    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/991b571e-8f93-47b0-9bb8-ab71ee26897c-home\") pod \"rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn\" (UID: \"991b571e-8f93-47b0-9bb8-ab71ee26897c\") " pod="ai-tenant-e2e-shared-a-w1-f7ee8a/rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn"
Oct 30 14:07:19.226757 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:19.226747    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/991b571e-8f93-47b0-9bb8-ab71ee26897c-model-cache\") pod \"rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn\" (UID: \"991b571e-8f93-47b0-9bb8-ab71ee26897c\") " pod="ai-tenant-e2e-shared-a-w1-f7ee8a/rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn"
Oct 30 14:07:19.226912 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:19.226888    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/991b571e-8f93-47b0-9bb8-ab71ee26897c-dshm\") pod \"rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn\" (UID: \"991b571e-8f93-47b0-9bb8-ab71ee26897c\") " pod="ai-tenant-e2e-shared-a-w1-f7ee8a/rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn"
Oct 30 14:07:19.227004 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:19.226955    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/991b571e-8f93-47b0-9bb8-ab71ee26897c-tmp-dir\") pod \"rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn\" (UID: \"991b571e-8f93-47b0-9bb8-ab71ee26897c\") " pod="ai-tenant-e2e-shared-a-w1-f7ee8a/rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn"
Oct 30 14:07:19.227065 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:19.227025    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/991b571e-8f93-47b0-9bb8-ab71ee26897c-tls-certs\") pod \"rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn\" (UID: \"991b571e-8f93-47b0-9bb8-ab71ee26897c\") " pod="ai-tenant-e2e-shared-a-w1-f7ee8a/rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn"
Oct 30 14:07:19.227495 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:19.227252    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/991b571e-8f93-47b0-9bb8-ab71ee26897c-home\") pod \"rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn\" (UID: \"991b571e-8f93-47b0-9bb8-ab71ee26897c\") " pod="ai-tenant-e2e-shared-a-w1-f7ee8a/rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn"
Oct 30 14:07:19.227495 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:19.227339    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/991b571e-8f93-47b0-9bb8-ab71ee26897c-model-cache\") pod \"rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn\" (UID: \"991b571e-8f93-47b0-9bb8-ab71ee26897c\") " pod="ai-tenant-e2e-shared-a-w1-f7ee8a/rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn"
Oct 30 14:07:19.227495 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:19.227445    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/991b571e-8f93-47b0-9bb8-ab71ee26897c-tmp-dir\") pod \"rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn\" (UID: \"991b571e-8f93-47b0-9bb8-ab71ee26897c\") " pod="ai-tenant-e2e-shared-a-w1-f7ee8a/rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn"
Oct 30 14:07:19.229947 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:19.229912    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/991b571e-8f93-47b0-9bb8-ab71ee26897c-dshm\") pod \"rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn\" (UID: \"991b571e-8f93-47b0-9bb8-ab71ee26897c\") " pod="ai-tenant-e2e-shared-a-w1-f7ee8a/rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn"
Oct 30 14:07:19.230178 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:19.230157    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/991b571e-8f93-47b0-9bb8-ab71ee26897c-tls-certs\") pod \"rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn\" (UID: \"991b571e-8f93-47b0-9bb8-ab71ee26897c\") " pod="ai-tenant-e2e-shared-a-w1-f7ee8a/rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn"
Oct 30 14:07:19.249681 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:19.249542    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgpws\" (UniqueName: \"kubernetes.io/projected/991b571e-8f93-47b0-9bb8-ab71ee26897c-kube-api-access-vgpws\") pod \"rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn\" (UID: \"991b571e-8f93-47b0-9bb8-ab71ee26897c\") " pod="ai-tenant-e2e-shared-a-w1-f7ee8a/rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn"
Oct 30 14:07:19.331464 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:19.331361    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-f7ee8a/rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn"
Oct 30 14:07:19.650163 ip-10-0-134-31 kubenswrapper[5093]: W1030 14:07:19.650112    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod991b571e_8f93_47b0_9bb8_ab71ee26897c.slice/crio-0c927295d96d5a35d0d55123d804b3a01ae19cd8529dc9fcd195b1df13ec0cd8 WatchSource:0}: Error finding container 0c927295d96d5a35d0d55123d804b3a01ae19cd8529dc9fcd195b1df13ec0cd8: Status 404 returned error can't find the container with id 0c927295d96d5a35d0d55123d804b3a01ae19cd8529dc9fcd195b1df13ec0cd8
Oct 30 14:07:19.652222 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:19.652188    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-f7ee8a/rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn"]
Oct 30 14:07:20.385570 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:20.385533    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-f7ee8a/rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn" event={"ID":"991b571e-8f93-47b0-9bb8-ab71ee26897c","Type":"ContainerStarted","Data":"c2062ed9ec3ecb88cc205985959891486b1d3dc8d242bcb15c8ee0b465e51479"}
Oct 30 14:07:20.385955 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:20.385577    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-f7ee8a/rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn" event={"ID":"991b571e-8f93-47b0-9bb8-ab71ee26897c","Type":"ContainerStarted","Data":"0c927295d96d5a35d0d55123d804b3a01ae19cd8529dc9fcd195b1df13ec0cd8"}
Oct 30 14:07:20.441215 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:20.441148    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-f7ee8a/rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn" podStartSLOduration=2.441119372 podStartE2EDuration="2.441119372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:07:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:07:20.438253904 +0000 UTC m=+3859.585521145" watchObservedRunningTime="2026-10-30 14:07:20.441119372 +0000 UTC m=+3859.588386614"
Oct 30 14:07:21.172962 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:07:21.172916    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 14:07:21.332281 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:21.332226    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-f7ee8a/rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn"
Oct 30 14:07:29.332673 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:29.332630    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-f7ee8a/rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn"
Oct 30 14:07:29.346662 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:29.346623    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-f7ee8a/rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn"
Oct 30 14:07:29.507958 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:29.507927    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-f7ee8a/rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn"
Oct 30 14:07:33.279973 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:33.279929    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt"]
Oct 30 14:07:38.937296 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:38.937248    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-5852ff-6759c64f6-5vvbb"]
Oct 30 14:07:38.969462 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:38.969429    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-5852ff-7899c595c5-2z2l5"]
Oct 30 14:07:43.938058 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:43.938002    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-5852ff-6759c64f6-5vvbb" podUID="3d17739e-fca2-4b2b-aa88-56f7d3a3399e" containerName="payload-processing" containerID="cri-o://9f8a03054677d3a0401e3f89aedfdb00bfa9ddad18e4fdf5a77875c8d2284d64" gracePeriod=25
Oct 30 14:07:43.970401 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:43.970357    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5852ff-7899c595c5-2z2l5" podUID="db92ac02-6454-4587-a1ec-f98b92646dfb" containerName="payload-pre-processing" containerID="cri-o://dad4d976b7c5af384d1a04fe02f4c2770a02b9c7d3f2b4f66a2c623ed240d7c3" gracePeriod=25
Oct 30 14:07:47.505085 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:47.505043    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-167965-6b479587d8-zktwq"]
Oct 30 14:07:47.541403 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:47.541360    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-167965-fdb89d64c-td5pc"]
Oct 30 14:07:48.302232 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:48.302181    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt" podUID="4ae52c69-0c30-438f-9ac0-c87b6de16061" containerName="main" containerID="cri-o://345309aba8f9a3c65e4b8614e3de3929be5b4858b91f8233b32dae4db5ac2bac" gracePeriod=45
Oct 30 14:07:48.579943 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:48.579910    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt"
Oct 30 14:07:48.660391 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:48.660339    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ae52c69-0c30-438f-9ac0-c87b6de16061-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ae52c69-0c30-438f-9ac0-c87b6de16061-dshm\") pod \"4ae52c69-0c30-438f-9ac0-c87b6de16061\" (UID: \"4ae52c69-0c30-438f-9ac0-c87b6de16061\") "
Oct 30 14:07:48.660652 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:48.660488    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ae52c69-0c30-438f-9ac0-c87b6de16061-home\" (UniqueName: \"kubernetes.io/empty-dir/4ae52c69-0c30-438f-9ac0-c87b6de16061-home\") pod \"4ae52c69-0c30-438f-9ac0-c87b6de16061\" (UID: \"4ae52c69-0c30-438f-9ac0-c87b6de16061\") "
Oct 30 14:07:48.660652 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:48.660542    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ae52c69-0c30-438f-9ac0-c87b6de16061-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ae52c69-0c30-438f-9ac0-c87b6de16061-tmp-dir\") pod \"4ae52c69-0c30-438f-9ac0-c87b6de16061\" (UID: \"4ae52c69-0c30-438f-9ac0-c87b6de16061\") "
Oct 30 14:07:48.660652 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:48.660574    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ae52c69-0c30-438f-9ac0-c87b6de16061-kube-api-access-8ltqf\" (UniqueName: \"kubernetes.io/projected/4ae52c69-0c30-438f-9ac0-c87b6de16061-kube-api-access-8ltqf\") pod \"4ae52c69-0c30-438f-9ac0-c87b6de16061\" (UID: \"4ae52c69-0c30-438f-9ac0-c87b6de16061\") "
Oct 30 14:07:48.660652 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:48.660604    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ae52c69-0c30-438f-9ac0-c87b6de16061-tls-certs\" (UniqueName: \"kubernetes.io/secret/4ae52c69-0c30-438f-9ac0-c87b6de16061-tls-certs\") pod \"4ae52c69-0c30-438f-9ac0-c87b6de16061\" (UID: \"4ae52c69-0c30-438f-9ac0-c87b6de16061\") "
Oct 30 14:07:48.660921 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:48.660675    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ae52c69-0c30-438f-9ac0-c87b6de16061-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ae52c69-0c30-438f-9ac0-c87b6de16061-model-cache\") pod \"4ae52c69-0c30-438f-9ac0-c87b6de16061\" (UID: \"4ae52c69-0c30-438f-9ac0-c87b6de16061\") "
Oct 30 14:07:48.660921 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:48.660838    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ae52c69-0c30-438f-9ac0-c87b6de16061-home" pod "4ae52c69-0c30-438f-9ac0-c87b6de16061" (UID: "4ae52c69-0c30-438f-9ac0-c87b6de16061"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:07:48.661368 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:48.661334    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ae52c69-0c30-438f-9ac0-c87b6de16061-model-cache" pod "4ae52c69-0c30-438f-9ac0-c87b6de16061" (UID: "4ae52c69-0c30-438f-9ac0-c87b6de16061"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:07:48.661368 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:48.661372    5093 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ae52c69-0c30-438f-9ac0-c87b6de16061-home\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:07:48.661587 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:48.661401    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ae52c69-0c30-438f-9ac0-c87b6de16061-tmp-dir" pod "4ae52c69-0c30-438f-9ac0-c87b6de16061" (UID: "4ae52c69-0c30-438f-9ac0-c87b6de16061"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:07:48.663179 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:48.662957    5093 generic.go:356] "Generic (PLEG): container finished" podID="4ae52c69-0c30-438f-9ac0-c87b6de16061" containerID="345309aba8f9a3c65e4b8614e3de3929be5b4858b91f8233b32dae4db5ac2bac" exitCode=0
Oct 30 14:07:48.663179 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:48.663012    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt" event={"ID":"4ae52c69-0c30-438f-9ac0-c87b6de16061","Type":"ContainerDied","Data":"345309aba8f9a3c65e4b8614e3de3929be5b4858b91f8233b32dae4db5ac2bac"}
Oct 30 14:07:48.663179 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:48.663040    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt"
Oct 30 14:07:48.663179 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:48.663060    5093 scope.go:122] "RemoveContainer" containerID="345309aba8f9a3c65e4b8614e3de3929be5b4858b91f8233b32dae4db5ac2bac"
Oct 30 14:07:48.663597 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:48.663047    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt" event={"ID":"4ae52c69-0c30-438f-9ac0-c87b6de16061","Type":"ContainerDied","Data":"1a11c3af534ae2a30fc8022fa68ed1702134bbc8f496cf00074a259911225fcc"}
Oct 30 14:07:48.678066 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:48.678035    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ae52c69-0c30-438f-9ac0-c87b6de16061-tls-certs" pod "4ae52c69-0c30-438f-9ac0-c87b6de16061" (UID: "4ae52c69-0c30-438f-9ac0-c87b6de16061"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:07:48.679596 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:48.679515    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ae52c69-0c30-438f-9ac0-c87b6de16061-dshm" pod "4ae52c69-0c30-438f-9ac0-c87b6de16061" (UID: "4ae52c69-0c30-438f-9ac0-c87b6de16061"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:07:48.691355 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:48.691333    5093 scope.go:122] "RemoveContainer" containerID="345309aba8f9a3c65e4b8614e3de3929be5b4858b91f8233b32dae4db5ac2bac"
Oct 30 14:07:48.691627 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:48.691599    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ae52c69-0c30-438f-9ac0-c87b6de16061-kube-api-access-8ltqf" pod "4ae52c69-0c30-438f-9ac0-c87b6de16061" (UID: "4ae52c69-0c30-438f-9ac0-c87b6de16061"). InnerVolumeSpecName "kube-api-access-8ltqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:07:48.691790 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:07:48.691760    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"345309aba8f9a3c65e4b8614e3de3929be5b4858b91f8233b32dae4db5ac2bac\": container with ID starting with 345309aba8f9a3c65e4b8614e3de3929be5b4858b91f8233b32dae4db5ac2bac not found: ID does not exist" containerID="345309aba8f9a3c65e4b8614e3de3929be5b4858b91f8233b32dae4db5ac2bac"
Oct 30 14:07:48.691888 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:48.691789    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"345309aba8f9a3c65e4b8614e3de3929be5b4858b91f8233b32dae4db5ac2bac"} err="failed to get container status \"345309aba8f9a3c65e4b8614e3de3929be5b4858b91f8233b32dae4db5ac2bac\": rpc error: code = NotFound desc = could not find container \"345309aba8f9a3c65e4b8614e3de3929be5b4858b91f8233b32dae4db5ac2bac\": container with ID starting with 345309aba8f9a3c65e4b8614e3de3929be5b4858b91f8233b32dae4db5ac2bac not found: ID does not exist"
Oct 30 14:07:48.762828 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:48.762787    5093 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ae52c69-0c30-438f-9ac0-c87b6de16061-tmp-dir\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:07:48.762828 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:48.762819    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ltqf\" (UniqueName: \"kubernetes.io/projected/4ae52c69-0c30-438f-9ac0-c87b6de16061-kube-api-access-8ltqf\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:07:48.762828 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:48.762831    5093 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ae52c69-0c30-438f-9ac0-c87b6de16061-tls-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:07:48.763065 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:48.762842    5093 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ae52c69-0c30-438f-9ac0-c87b6de16061-model-cache\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:07:48.763065 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:48.762850    5093 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ae52c69-0c30-438f-9ac0-c87b6de16061-dshm\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:07:49.013151 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:49.013093    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt"]
Oct 30 14:07:49.016478 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:49.016453    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-5852ff/premium-w3-5852ff-kserve-5c95746977-klqrt"]
Oct 30 14:07:51.004653 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:51.004610    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ae52c69-0c30-438f-9ac0-c87b6de16061" path="/var/lib/kubelet/pods/4ae52c69-0c30-438f-9ac0-c87b6de16061/volumes"
Oct 30 14:07:51.169975 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:07:51.169931    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 14:07:52.505927 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:52.505875    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-167965-6b479587d8-zktwq" podUID="1f39b9a2-0b8a-4eb7-9e7d-9a90d357672c" containerName="payload-processing" containerID="cri-o://659d5097d58c262f7a47798d916ca7756cd69256ba8fcb213cb859a3efd6b19c" gracePeriod=25
Oct 30 14:07:52.542528 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:52.542477    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-167965-fdb89d64c-td5pc" podUID="27a0b223-6a47-4bf9-8f6e-dd5a8bc80912" containerName="payload-pre-processing" containerID="cri-o://a93d067852c3eccf13e72988bbb4992964af2be0f59297e5fa88dafcf7305a49" gracePeriod=25
Oct 30 14:07:52.697940 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:52.697720    5093 generic.go:356] "Generic (PLEG): container finished" podID="27a0b223-6a47-4bf9-8f6e-dd5a8bc80912" containerID="a93d067852c3eccf13e72988bbb4992964af2be0f59297e5fa88dafcf7305a49" exitCode=0
Oct 30 14:07:52.697940 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:52.697878    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-167965-fdb89d64c-td5pc" event={"ID":"27a0b223-6a47-4bf9-8f6e-dd5a8bc80912","Type":"ContainerDied","Data":"a93d067852c3eccf13e72988bbb4992964af2be0f59297e5fa88dafcf7305a49"}
Oct 30 14:07:52.700164 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:52.700097    5093 generic.go:356] "Generic (PLEG): container finished" podID="1f39b9a2-0b8a-4eb7-9e7d-9a90d357672c" containerID="659d5097d58c262f7a47798d916ca7756cd69256ba8fcb213cb859a3efd6b19c" exitCode=0
Oct 30 14:07:52.700313 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:52.700195    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-167965-6b479587d8-zktwq" event={"ID":"1f39b9a2-0b8a-4eb7-9e7d-9a90d357672c","Type":"ContainerDied","Data":"659d5097d58c262f7a47798d916ca7756cd69256ba8fcb213cb859a3efd6b19c"}
Oct 30 14:07:52.845856 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:52.845822    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-167965-6b479587d8-zktwq"
Oct 30 14:07:52.883895 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:52.883866    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-167965-fdb89d64c-td5pc"
Oct 30 14:07:53.012398 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:53.012367    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f39b9a2-0b8a-4eb7-9e7d-9a90d357672c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f39b9a2-0b8a-4eb7-9e7d-9a90d357672c-plugins-config-volume\") pod \"1f39b9a2-0b8a-4eb7-9e7d-9a90d357672c\" (UID: \"1f39b9a2-0b8a-4eb7-9e7d-9a90d357672c\") "
Oct 30 14:07:53.012585 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:53.012472    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27a0b223-6a47-4bf9-8f6e-dd5a8bc80912-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27a0b223-6a47-4bf9-8f6e-dd5a8bc80912-plugins-config-volume\") pod \"27a0b223-6a47-4bf9-8f6e-dd5a8bc80912\" (UID: \"27a0b223-6a47-4bf9-8f6e-dd5a8bc80912\") "
Oct 30 14:07:53.012632 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:53.012600    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27a0b223-6a47-4bf9-8f6e-dd5a8bc80912-kube-api-access-7jf2v\" (UniqueName: \"kubernetes.io/projected/27a0b223-6a47-4bf9-8f6e-dd5a8bc80912-kube-api-access-7jf2v\") pod \"27a0b223-6a47-4bf9-8f6e-dd5a8bc80912\" (UID: \"27a0b223-6a47-4bf9-8f6e-dd5a8bc80912\") "
Oct 30 14:07:53.012670 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:53.012659    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f39b9a2-0b8a-4eb7-9e7d-9a90d357672c-kube-api-access-njzz8\" (UniqueName: \"kubernetes.io/projected/1f39b9a2-0b8a-4eb7-9e7d-9a90d357672c-kube-api-access-njzz8\") pod \"1f39b9a2-0b8a-4eb7-9e7d-9a90d357672c\" (UID: \"1f39b9a2-0b8a-4eb7-9e7d-9a90d357672c\") "
Oct 30 14:07:53.012839 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:53.012809    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f39b9a2-0b8a-4eb7-9e7d-9a90d357672c-plugins-config-volume" pod "1f39b9a2-0b8a-4eb7-9e7d-9a90d357672c" (UID: "1f39b9a2-0b8a-4eb7-9e7d-9a90d357672c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:07:53.012929 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:53.012840    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27a0b223-6a47-4bf9-8f6e-dd5a8bc80912-plugins-config-volume" pod "27a0b223-6a47-4bf9-8f6e-dd5a8bc80912" (UID: "27a0b223-6a47-4bf9-8f6e-dd5a8bc80912"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:07:53.013034 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:53.013015    5093 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27a0b223-6a47-4bf9-8f6e-dd5a8bc80912-plugins-config-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:07:53.013077 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:53.013037    5093 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f39b9a2-0b8a-4eb7-9e7d-9a90d357672c-plugins-config-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:07:53.014760 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:53.014740    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27a0b223-6a47-4bf9-8f6e-dd5a8bc80912-kube-api-access-7jf2v" pod "27a0b223-6a47-4bf9-8f6e-dd5a8bc80912" (UID: "27a0b223-6a47-4bf9-8f6e-dd5a8bc80912"). InnerVolumeSpecName "kube-api-access-7jf2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:07:53.014831 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:53.014761    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f39b9a2-0b8a-4eb7-9e7d-9a90d357672c-kube-api-access-njzz8" pod "1f39b9a2-0b8a-4eb7-9e7d-9a90d357672c" (UID: "1f39b9a2-0b8a-4eb7-9e7d-9a90d357672c"). InnerVolumeSpecName "kube-api-access-njzz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:07:53.114387 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:53.114299    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jf2v\" (UniqueName: \"kubernetes.io/projected/27a0b223-6a47-4bf9-8f6e-dd5a8bc80912-kube-api-access-7jf2v\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:07:53.114387 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:53.114333    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njzz8\" (UniqueName: \"kubernetes.io/projected/1f39b9a2-0b8a-4eb7-9e7d-9a90d357672c-kube-api-access-njzz8\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:07:53.640789 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:53.640746    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-5852ff-6759c64f6-5vvbb" podUID="3d17739e-fca2-4b2b-aa88-56f7d3a3399e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:07:53.641224 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:53.640791    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5852ff-7899c595c5-2z2l5" podUID="db92ac02-6454-4587-a1ec-f98b92646dfb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:07:53.708867 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:53.708834    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-167965-fdb89d64c-td5pc"
Oct 30 14:07:53.709069 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:53.708868    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-167965-fdb89d64c-td5pc" event={"ID":"27a0b223-6a47-4bf9-8f6e-dd5a8bc80912","Type":"ContainerDied","Data":"a9afdf01cefad36492f495bfd32267fcb0376a6b9ca0f1edab449e2dd60e163e"}
Oct 30 14:07:53.709069 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:53.708944    5093 scope.go:122] "RemoveContainer" containerID="a93d067852c3eccf13e72988bbb4992964af2be0f59297e5fa88dafcf7305a49"
Oct 30 14:07:53.710487 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:53.710461    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-167965-6b479587d8-zktwq" event={"ID":"1f39b9a2-0b8a-4eb7-9e7d-9a90d357672c","Type":"ContainerDied","Data":"1e51eabfef65de87f4e697a52b30995162ff67b1cfdf5877e90c6d0e583d6b11"}
Oct 30 14:07:53.710602 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:53.710557    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-167965-6b479587d8-zktwq"
Oct 30 14:07:53.720839 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:53.720817    5093 scope.go:122] "RemoveContainer" containerID="659d5097d58c262f7a47798d916ca7756cd69256ba8fcb213cb859a3efd6b19c"
Oct 30 14:07:53.774592 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:53.774557    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-167965-6b479587d8-zktwq"]
Oct 30 14:07:53.787700 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:53.787636    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-167965-6b479587d8-zktwq"]
Oct 30 14:07:53.822113 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:53.822081    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-167965-fdb89d64c-td5pc"]
Oct 30 14:07:53.835633 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:53.835595    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-167965-fdb89d64c-td5pc"]
Oct 30 14:07:55.000060 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:55.000023    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f39b9a2-0b8a-4eb7-9e7d-9a90d357672c" path="/var/lib/kubelet/pods/1f39b9a2-0b8a-4eb7-9e7d-9a90d357672c/volumes"
Oct 30 14:07:55.000518 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:07:55.000502    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27a0b223-6a47-4bf9-8f6e-dd5a8bc80912" path="/var/lib/kubelet/pods/27a0b223-6a47-4bf9-8f6e-dd5a8bc80912/volumes"
Oct 30 14:08:01.872861 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:01.872822    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jdj4_f93ee017-ab79-447c-90e0-5b85b71dba24/console-operator/1.log"
Oct 30 14:08:01.879326 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:01.879299    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tnm_3a675ac7-fbe1-449b-accf-bb0456ed0bd3/ovn-acl-logging/0.log"
Oct 30 14:08:01.883459 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:01.883428    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jdj4_f93ee017-ab79-447c-90e0-5b85b71dba24/console-operator/1.log"
Oct 30 14:08:01.889800 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:01.889775    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tnm_3a675ac7-fbe1-449b-accf-bb0456ed0bd3/ovn-acl-logging/0.log"
Oct 30 14:08:03.641338 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:03.641296    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5852ff-7899c595c5-2z2l5" podUID="db92ac02-6454-4587-a1ec-f98b92646dfb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:08:03.641854 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:03.641296    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-5852ff-6759c64f6-5vvbb" podUID="3d17739e-fca2-4b2b-aa88-56f7d3a3399e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:08:04.265668 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:04.265628    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-5852ff-6759c64f6-5vvbb"
Oct 30 14:08:04.305222 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:04.305194    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5852ff-7899c595c5-2z2l5"
Oct 30 14:08:04.382565 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:04.382529    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db92ac02-6454-4587-a1ec-f98b92646dfb-kube-api-access-8bndb\" (UniqueName: \"kubernetes.io/projected/db92ac02-6454-4587-a1ec-f98b92646dfb-kube-api-access-8bndb\") pod \"db92ac02-6454-4587-a1ec-f98b92646dfb\" (UID: \"db92ac02-6454-4587-a1ec-f98b92646dfb\") "
Oct 30 14:08:04.382771 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:04.382594    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d17739e-fca2-4b2b-aa88-56f7d3a3399e-kube-api-access-c2hjn\" (UniqueName: \"kubernetes.io/projected/3d17739e-fca2-4b2b-aa88-56f7d3a3399e-kube-api-access-c2hjn\") pod \"3d17739e-fca2-4b2b-aa88-56f7d3a3399e\" (UID: \"3d17739e-fca2-4b2b-aa88-56f7d3a3399e\") "
Oct 30 14:08:04.382771 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:04.382677    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db92ac02-6454-4587-a1ec-f98b92646dfb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db92ac02-6454-4587-a1ec-f98b92646dfb-plugins-config-volume\") pod \"db92ac02-6454-4587-a1ec-f98b92646dfb\" (UID: \"db92ac02-6454-4587-a1ec-f98b92646dfb\") "
Oct 30 14:08:04.382771 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:04.382726    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d17739e-fca2-4b2b-aa88-56f7d3a3399e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d17739e-fca2-4b2b-aa88-56f7d3a3399e-plugins-config-volume\") pod \"3d17739e-fca2-4b2b-aa88-56f7d3a3399e\" (UID: \"3d17739e-fca2-4b2b-aa88-56f7d3a3399e\") "
Oct 30 14:08:04.383348 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:04.383312    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db92ac02-6454-4587-a1ec-f98b92646dfb-plugins-config-volume" pod "db92ac02-6454-4587-a1ec-f98b92646dfb" (UID: "db92ac02-6454-4587-a1ec-f98b92646dfb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:08:04.383541 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:04.383409    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d17739e-fca2-4b2b-aa88-56f7d3a3399e-plugins-config-volume" pod "3d17739e-fca2-4b2b-aa88-56f7d3a3399e" (UID: "3d17739e-fca2-4b2b-aa88-56f7d3a3399e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:08:04.383881 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:04.383860    5093 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db92ac02-6454-4587-a1ec-f98b92646dfb-plugins-config-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:08:04.383881 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:04.383882    5093 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d17739e-fca2-4b2b-aa88-56f7d3a3399e-plugins-config-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:08:04.389390 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:04.389354    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d17739e-fca2-4b2b-aa88-56f7d3a3399e-kube-api-access-c2hjn" pod "3d17739e-fca2-4b2b-aa88-56f7d3a3399e" (UID: "3d17739e-fca2-4b2b-aa88-56f7d3a3399e"). InnerVolumeSpecName "kube-api-access-c2hjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:08:04.389561 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:04.389415    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db92ac02-6454-4587-a1ec-f98b92646dfb-kube-api-access-8bndb" pod "db92ac02-6454-4587-a1ec-f98b92646dfb" (UID: "db92ac02-6454-4587-a1ec-f98b92646dfb"). InnerVolumeSpecName "kube-api-access-8bndb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:08:04.484831 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:04.484732    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bndb\" (UniqueName: \"kubernetes.io/projected/db92ac02-6454-4587-a1ec-f98b92646dfb-kube-api-access-8bndb\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:08:04.484831 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:04.484776    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2hjn\" (UniqueName: \"kubernetes.io/projected/3d17739e-fca2-4b2b-aa88-56f7d3a3399e-kube-api-access-c2hjn\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:08:04.808275 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:04.808180    5093 generic.go:356] "Generic (PLEG): container finished" podID="db92ac02-6454-4587-a1ec-f98b92646dfb" containerID="dad4d976b7c5af384d1a04fe02f4c2770a02b9c7d3f2b4f66a2c623ed240d7c3" exitCode=0
Oct 30 14:08:04.808275 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:04.808229    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5852ff-7899c595c5-2z2l5"
Oct 30 14:08:04.808762 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:04.808278    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5852ff-7899c595c5-2z2l5" event={"ID":"db92ac02-6454-4587-a1ec-f98b92646dfb","Type":"ContainerDied","Data":"dad4d976b7c5af384d1a04fe02f4c2770a02b9c7d3f2b4f66a2c623ed240d7c3"}
Oct 30 14:08:04.808762 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:04.808347    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5852ff-7899c595c5-2z2l5" event={"ID":"db92ac02-6454-4587-a1ec-f98b92646dfb","Type":"ContainerDied","Data":"4791f34f3edaadf526afadf52547c4fa923bc0b3565ed311548bb00110e04304"}
Oct 30 14:08:04.808762 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:04.808379    5093 scope.go:122] "RemoveContainer" containerID="dad4d976b7c5af384d1a04fe02f4c2770a02b9c7d3f2b4f66a2c623ed240d7c3"
Oct 30 14:08:04.810045 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:04.810017    5093 generic.go:356] "Generic (PLEG): container finished" podID="3d17739e-fca2-4b2b-aa88-56f7d3a3399e" containerID="9f8a03054677d3a0401e3f89aedfdb00bfa9ddad18e4fdf5a77875c8d2284d64" exitCode=0
Oct 30 14:08:04.810223 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:04.810080    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-5852ff-6759c64f6-5vvbb"
Oct 30 14:08:04.810223 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:04.810158    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-5852ff-6759c64f6-5vvbb" event={"ID":"3d17739e-fca2-4b2b-aa88-56f7d3a3399e","Type":"ContainerDied","Data":"9f8a03054677d3a0401e3f89aedfdb00bfa9ddad18e4fdf5a77875c8d2284d64"}
Oct 30 14:08:04.810223 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:04.810211    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-5852ff-6759c64f6-5vvbb" event={"ID":"3d17739e-fca2-4b2b-aa88-56f7d3a3399e","Type":"ContainerDied","Data":"1c283ceff9e109949f0fc2c9d7be795b7b3cee1393ed08561afad42681a0d08c"}
Oct 30 14:08:04.824028 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:04.824001    5093 scope.go:122] "RemoveContainer" containerID="dad4d976b7c5af384d1a04fe02f4c2770a02b9c7d3f2b4f66a2c623ed240d7c3"
Oct 30 14:08:04.824629 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:08:04.824605    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dad4d976b7c5af384d1a04fe02f4c2770a02b9c7d3f2b4f66a2c623ed240d7c3\": container with ID starting with dad4d976b7c5af384d1a04fe02f4c2770a02b9c7d3f2b4f66a2c623ed240d7c3 not found: ID does not exist" containerID="dad4d976b7c5af384d1a04fe02f4c2770a02b9c7d3f2b4f66a2c623ed240d7c3"
Oct 30 14:08:04.824748 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:04.824637    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dad4d976b7c5af384d1a04fe02f4c2770a02b9c7d3f2b4f66a2c623ed240d7c3"} err="failed to get container status \"dad4d976b7c5af384d1a04fe02f4c2770a02b9c7d3f2b4f66a2c623ed240d7c3\": rpc error: code = NotFound desc = could not find container \"dad4d976b7c5af384d1a04fe02f4c2770a02b9c7d3f2b4f66a2c623ed240d7c3\": container with ID starting with dad4d976b7c5af384d1a04fe02f4c2770a02b9c7d3f2b4f66a2c623ed240d7c3 not found: ID does not exist"
Oct 30 14:08:04.824748 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:04.824655    5093 scope.go:122] "RemoveContainer" containerID="9f8a03054677d3a0401e3f89aedfdb00bfa9ddad18e4fdf5a77875c8d2284d64"
Oct 30 14:08:04.834675 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:04.834646    5093 scope.go:122] "RemoveContainer" containerID="9f8a03054677d3a0401e3f89aedfdb00bfa9ddad18e4fdf5a77875c8d2284d64"
Oct 30 14:08:04.834962 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:08:04.834940    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f8a03054677d3a0401e3f89aedfdb00bfa9ddad18e4fdf5a77875c8d2284d64\": container with ID starting with 9f8a03054677d3a0401e3f89aedfdb00bfa9ddad18e4fdf5a77875c8d2284d64 not found: ID does not exist" containerID="9f8a03054677d3a0401e3f89aedfdb00bfa9ddad18e4fdf5a77875c8d2284d64"
Oct 30 14:08:04.835030 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:04.834969    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f8a03054677d3a0401e3f89aedfdb00bfa9ddad18e4fdf5a77875c8d2284d64"} err="failed to get container status \"9f8a03054677d3a0401e3f89aedfdb00bfa9ddad18e4fdf5a77875c8d2284d64\": rpc error: code = NotFound desc = could not find container \"9f8a03054677d3a0401e3f89aedfdb00bfa9ddad18e4fdf5a77875c8d2284d64\": container with ID starting with 9f8a03054677d3a0401e3f89aedfdb00bfa9ddad18e4fdf5a77875c8d2284d64 not found: ID does not exist"
Oct 30 14:08:04.862891 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:04.862859    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-5852ff-6759c64f6-5vvbb"]
Oct 30 14:08:04.877260 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:04.877225    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-5852ff-6759c64f6-5vvbb"]
Oct 30 14:08:04.905438 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:04.905398    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-5852ff-7899c595c5-2z2l5"]
Oct 30 14:08:04.910974 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:04.910938    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-5852ff-7899c595c5-2z2l5"]
Oct 30 14:08:05.000587 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:05.000555    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d17739e-fca2-4b2b-aa88-56f7d3a3399e" path="/var/lib/kubelet/pods/3d17739e-fca2-4b2b-aa88-56f7d3a3399e/volumes"
Oct 30 14:08:05.001050 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:05.001035    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db92ac02-6454-4587-a1ec-f98b92646dfb" path="/var/lib/kubelet/pods/db92ac02-6454-4587-a1ec-f98b92646dfb/volumes"
Oct 30 14:08:10.664897 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:10.656361    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-167965-74ddd66c88-rw9pw"]
Oct 30 14:08:10.664897 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:10.656654    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-167965-74ddd66c88-rw9pw" podUID="17e5bb80-effa-4c06-b94f-b7e62b372ea8" containerName="maas-api" containerID="cri-o://84c3cb33553972cdbbb778d577d575dc540fe57b9809880e00a72ec79b3455b9" gracePeriod=30
Oct 30 14:08:10.865935 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:10.865893    5093 generic.go:356] "Generic (PLEG): container finished" podID="17e5bb80-effa-4c06-b94f-b7e62b372ea8" containerID="84c3cb33553972cdbbb778d577d575dc540fe57b9809880e00a72ec79b3455b9" exitCode=0
Oct 30 14:08:10.866174 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:10.866073    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-167965-74ddd66c88-rw9pw" event={"ID":"17e5bb80-effa-4c06-b94f-b7e62b372ea8","Type":"ContainerDied","Data":"84c3cb33553972cdbbb778d577d575dc540fe57b9809880e00a72ec79b3455b9"}
Oct 30 14:08:10.970475 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:10.970385    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-167965-74ddd66c88-rw9pw"
Oct 30 14:08:11.077215 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:11.077176    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17e5bb80-effa-4c06-b94f-b7e62b372ea8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/17e5bb80-effa-4c06-b94f-b7e62b372ea8-metrics-certs\") pod \"17e5bb80-effa-4c06-b94f-b7e62b372ea8\" (UID: \"17e5bb80-effa-4c06-b94f-b7e62b372ea8\") "
Oct 30 14:08:11.077423 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:11.077232    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17e5bb80-effa-4c06-b94f-b7e62b372ea8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17e5bb80-effa-4c06-b94f-b7e62b372ea8-maas-api-tls\") pod \"17e5bb80-effa-4c06-b94f-b7e62b372ea8\" (UID: \"17e5bb80-effa-4c06-b94f-b7e62b372ea8\") "
Oct 30 14:08:11.077423 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:11.077367    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17e5bb80-effa-4c06-b94f-b7e62b372ea8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17e5bb80-effa-4c06-b94f-b7e62b372ea8-openshift-service-ca\") pod \"17e5bb80-effa-4c06-b94f-b7e62b372ea8\" (UID: \"17e5bb80-effa-4c06-b94f-b7e62b372ea8\") "
Oct 30 14:08:11.077565 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:11.077546    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17e5bb80-effa-4c06-b94f-b7e62b372ea8-kube-api-access-54rtk\" (UniqueName: \"kubernetes.io/projected/17e5bb80-effa-4c06-b94f-b7e62b372ea8-kube-api-access-54rtk\") pod \"17e5bb80-effa-4c06-b94f-b7e62b372ea8\" (UID: \"17e5bb80-effa-4c06-b94f-b7e62b372ea8\") "
Oct 30 14:08:11.077747 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:11.077721    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17e5bb80-effa-4c06-b94f-b7e62b372ea8-openshift-service-ca" pod "17e5bb80-effa-4c06-b94f-b7e62b372ea8" (UID: "17e5bb80-effa-4c06-b94f-b7e62b372ea8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:08:11.078036 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:11.078013    5093 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17e5bb80-effa-4c06-b94f-b7e62b372ea8-openshift-service-ca\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:08:11.079500 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:11.079476    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17e5bb80-effa-4c06-b94f-b7e62b372ea8-metrics-certs" pod "17e5bb80-effa-4c06-b94f-b7e62b372ea8" (UID: "17e5bb80-effa-4c06-b94f-b7e62b372ea8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:08:11.080016 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:11.079993    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17e5bb80-effa-4c06-b94f-b7e62b372ea8-maas-api-tls" pod "17e5bb80-effa-4c06-b94f-b7e62b372ea8" (UID: "17e5bb80-effa-4c06-b94f-b7e62b372ea8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:08:11.080116 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:11.080002    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17e5bb80-effa-4c06-b94f-b7e62b372ea8-kube-api-access-54rtk" pod "17e5bb80-effa-4c06-b94f-b7e62b372ea8" (UID: "17e5bb80-effa-4c06-b94f-b7e62b372ea8"). InnerVolumeSpecName "kube-api-access-54rtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:08:11.179482 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:11.179446    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54rtk\" (UniqueName: \"kubernetes.io/projected/17e5bb80-effa-4c06-b94f-b7e62b372ea8-kube-api-access-54rtk\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:08:11.179482 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:11.179479    5093 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17e5bb80-effa-4c06-b94f-b7e62b372ea8-metrics-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:08:11.179482 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:11.179488    5093 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17e5bb80-effa-4c06-b94f-b7e62b372ea8-maas-api-tls\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:08:11.876263 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:11.876227    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-167965-74ddd66c88-rw9pw"
Oct 30 14:08:11.876799 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:11.876227    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-167965-74ddd66c88-rw9pw" event={"ID":"17e5bb80-effa-4c06-b94f-b7e62b372ea8","Type":"ContainerDied","Data":"b3f50f9455f1904819d705550ca3d918e4051b368d52276bc1c9d4e9190aeb7e"}
Oct 30 14:08:11.876799 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:11.876388    5093 scope.go:122] "RemoveContainer" containerID="84c3cb33553972cdbbb778d577d575dc540fe57b9809880e00a72ec79b3455b9"
Oct 30 14:08:11.933413 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:11.933368    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-167965-74ddd66c88-rw9pw"]
Oct 30 14:08:11.938588 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:11.938561    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-167965-74ddd66c88-rw9pw"]
Oct 30 14:08:12.999771 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:12.999732    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17e5bb80-effa-4c06-b94f-b7e62b372ea8" path="/var/lib/kubelet/pods/17e5bb80-effa-4c06-b94f-b7e62b372ea8/volumes"
Oct 30 14:08:21.169729 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:08:21.169678    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 14:08:26.408667 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.408631    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6"]
Oct 30 14:08:26.409476 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.409452    5093 cpu_manager.go:414] "containerMap: removing container" podUID="db92ac02-6454-4587-a1ec-f98b92646dfb" containerName="payload-pre-processing"
Oct 30 14:08:26.409549 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.409480    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db92ac02-6454-4587-a1ec-f98b92646dfb" containerName="payload-pre-processing"
Oct 30 14:08:26.409549 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.409517    5093 cpu_manager.go:414] "containerMap: removing container" podUID="17e5bb80-effa-4c06-b94f-b7e62b372ea8" containerName="maas-api"
Oct 30 14:08:26.409549 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.409526    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17e5bb80-effa-4c06-b94f-b7e62b372ea8" containerName="maas-api"
Oct 30 14:08:26.409549 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.409544    5093 cpu_manager.go:414] "containerMap: removing container" podUID="4ae52c69-0c30-438f-9ac0-c87b6de16061" containerName="main"
Oct 30 14:08:26.409549 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.409550    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ae52c69-0c30-438f-9ac0-c87b6de16061" containerName="main"
Oct 30 14:08:26.409784 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.409571    5093 cpu_manager.go:414] "containerMap: removing container" podUID="18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e" containerName="istio-proxy"
Oct 30 14:08:26.409784 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.409579    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e" containerName="istio-proxy"
Oct 30 14:08:26.409784 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.409598    5093 cpu_manager.go:414] "containerMap: removing container" podUID="3d17739e-fca2-4b2b-aa88-56f7d3a3399e" containerName="payload-processing"
Oct 30 14:08:26.409784 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.409603    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d17739e-fca2-4b2b-aa88-56f7d3a3399e" containerName="payload-processing"
Oct 30 14:08:26.409784 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.409611    5093 cpu_manager.go:414] "containerMap: removing container" podUID="cc75f0e6-64c6-4fde-9476-3859350da986" containerName="istio-proxy"
Oct 30 14:08:26.409784 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.409615    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc75f0e6-64c6-4fde-9476-3859350da986" containerName="istio-proxy"
Oct 30 14:08:26.409784 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.409625    5093 cpu_manager.go:414] "containerMap: removing container" podUID="1f39b9a2-0b8a-4eb7-9e7d-9a90d357672c" containerName="payload-processing"
Oct 30 14:08:26.409784 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.409630    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f39b9a2-0b8a-4eb7-9e7d-9a90d357672c" containerName="payload-processing"
Oct 30 14:08:26.409784 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.409641    5093 cpu_manager.go:414] "containerMap: removing container" podUID="27a0b223-6a47-4bf9-8f6e-dd5a8bc80912" containerName="payload-pre-processing"
Oct 30 14:08:26.409784 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.409645    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27a0b223-6a47-4bf9-8f6e-dd5a8bc80912" containerName="payload-pre-processing"
Oct 30 14:08:26.409784 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.409774    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="db92ac02-6454-4587-a1ec-f98b92646dfb" containerName="payload-pre-processing"
Oct 30 14:08:26.410284 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.409798    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="17e5bb80-effa-4c06-b94f-b7e62b372ea8" containerName="maas-api"
Oct 30 14:08:26.410284 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.409811    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ae52c69-0c30-438f-9ac0-c87b6de16061" containerName="main"
Oct 30 14:08:26.410284 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.409821    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="18b2b0a1-af31-4cb1-acdf-6357fbf0ed7e" containerName="istio-proxy"
Oct 30 14:08:26.410284 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.409828    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d17739e-fca2-4b2b-aa88-56f7d3a3399e" containerName="payload-processing"
Oct 30 14:08:26.410284 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.409834    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc75f0e6-64c6-4fde-9476-3859350da986" containerName="istio-proxy"
Oct 30 14:08:26.410284 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.409842    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f39b9a2-0b8a-4eb7-9e7d-9a90d357672c" containerName="payload-processing"
Oct 30 14:08:26.410284 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.409848    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="27a0b223-6a47-4bf9-8f6e-dd5a8bc80912" containerName="payload-pre-processing"
Oct 30 14:08:26.415253 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.415233    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6"
Oct 30 14:08:26.439284 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.439254    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6"]
Oct 30 14:08:26.446728 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.446693    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-29a3f3-openshift-default-dockercfg-2sfrp\""
Oct 30 14:08:26.510992 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.510960    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e378bf44-e323-460f-9096-fd8835ad91b8-credential-socket\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6\" (UID: \"e378bf44-e323-460f-9096-fd8835ad91b8\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6"
Oct 30 14:08:26.511202 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.511017    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw2l8\" (UniqueName: \"kubernetes.io/projected/e378bf44-e323-460f-9096-fd8835ad91b8-kube-api-access-zw2l8\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6\" (UID: \"e378bf44-e323-460f-9096-fd8835ad91b8\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6"
Oct 30 14:08:26.511202 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.511140    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e378bf44-e323-460f-9096-fd8835ad91b8-istiod-ca-cert\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6\" (UID: \"e378bf44-e323-460f-9096-fd8835ad91b8\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6"
Oct 30 14:08:26.511316 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.511206    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e378bf44-e323-460f-9096-fd8835ad91b8-workload-certs\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6\" (UID: \"e378bf44-e323-460f-9096-fd8835ad91b8\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6"
Oct 30 14:08:26.511316 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.511297    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e378bf44-e323-460f-9096-fd8835ad91b8-istio-data\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6\" (UID: \"e378bf44-e323-460f-9096-fd8835ad91b8\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6"
Oct 30 14:08:26.511405 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.511340    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e378bf44-e323-460f-9096-fd8835ad91b8-istio-podinfo\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6\" (UID: \"e378bf44-e323-460f-9096-fd8835ad91b8\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6"
Oct 30 14:08:26.511541 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.511512    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e378bf44-e323-460f-9096-fd8835ad91b8-istio-envoy\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6\" (UID: \"e378bf44-e323-460f-9096-fd8835ad91b8\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6"
Oct 30 14:08:26.511625 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.511563    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e378bf44-e323-460f-9096-fd8835ad91b8-workload-socket\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6\" (UID: \"e378bf44-e323-460f-9096-fd8835ad91b8\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6"
Oct 30 14:08:26.511704 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.511678    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e378bf44-e323-460f-9096-fd8835ad91b8-istio-token\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6\" (UID: \"e378bf44-e323-460f-9096-fd8835ad91b8\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6"
Oct 30 14:08:26.612496 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.612456    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e378bf44-e323-460f-9096-fd8835ad91b8-istio-token\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6\" (UID: \"e378bf44-e323-460f-9096-fd8835ad91b8\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6"
Oct 30 14:08:26.612707 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.612591    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e378bf44-e323-460f-9096-fd8835ad91b8-credential-socket\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6\" (UID: \"e378bf44-e323-460f-9096-fd8835ad91b8\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6"
Oct 30 14:08:26.612707 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.612635    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw2l8\" (UniqueName: \"kubernetes.io/projected/e378bf44-e323-460f-9096-fd8835ad91b8-kube-api-access-zw2l8\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6\" (UID: \"e378bf44-e323-460f-9096-fd8835ad91b8\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6"
Oct 30 14:08:26.612707 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.612661    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e378bf44-e323-460f-9096-fd8835ad91b8-istiod-ca-cert\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6\" (UID: \"e378bf44-e323-460f-9096-fd8835ad91b8\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6"
Oct 30 14:08:26.612707 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.612701    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e378bf44-e323-460f-9096-fd8835ad91b8-workload-certs\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6\" (UID: \"e378bf44-e323-460f-9096-fd8835ad91b8\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6"
Oct 30 14:08:26.612940 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.612749    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e378bf44-e323-460f-9096-fd8835ad91b8-istio-data\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6\" (UID: \"e378bf44-e323-460f-9096-fd8835ad91b8\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6"
Oct 30 14:08:26.612940 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.612780    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e378bf44-e323-460f-9096-fd8835ad91b8-istio-podinfo\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6\" (UID: \"e378bf44-e323-460f-9096-fd8835ad91b8\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6"
Oct 30 14:08:26.612940 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.612900    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e378bf44-e323-460f-9096-fd8835ad91b8-istio-envoy\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6\" (UID: \"e378bf44-e323-460f-9096-fd8835ad91b8\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6"
Oct 30 14:08:26.612940 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.612934    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e378bf44-e323-460f-9096-fd8835ad91b8-workload-socket\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6\" (UID: \"e378bf44-e323-460f-9096-fd8835ad91b8\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6"
Oct 30 14:08:26.613155 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.613039    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e378bf44-e323-460f-9096-fd8835ad91b8-credential-socket\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6\" (UID: \"e378bf44-e323-460f-9096-fd8835ad91b8\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6"
Oct 30 14:08:26.613229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.613158    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e378bf44-e323-460f-9096-fd8835ad91b8-workload-certs\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6\" (UID: \"e378bf44-e323-460f-9096-fd8835ad91b8\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6"
Oct 30 14:08:26.613395 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.613371    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e378bf44-e323-460f-9096-fd8835ad91b8-workload-socket\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6\" (UID: \"e378bf44-e323-460f-9096-fd8835ad91b8\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6"
Oct 30 14:08:26.613525 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.613502    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e378bf44-e323-460f-9096-fd8835ad91b8-istiod-ca-cert\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6\" (UID: \"e378bf44-e323-460f-9096-fd8835ad91b8\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6"
Oct 30 14:08:26.613634 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.613609    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e378bf44-e323-460f-9096-fd8835ad91b8-istio-data\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6\" (UID: \"e378bf44-e323-460f-9096-fd8835ad91b8\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6"
Oct 30 14:08:26.615510 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.615479    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e378bf44-e323-460f-9096-fd8835ad91b8-istio-envoy\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6\" (UID: \"e378bf44-e323-460f-9096-fd8835ad91b8\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6"
Oct 30 14:08:26.615658 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.615633    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e378bf44-e323-460f-9096-fd8835ad91b8-istio-podinfo\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6\" (UID: \"e378bf44-e323-460f-9096-fd8835ad91b8\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6"
Oct 30 14:08:26.638221 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.633890    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e378bf44-e323-460f-9096-fd8835ad91b8-istio-token\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6\" (UID: \"e378bf44-e323-460f-9096-fd8835ad91b8\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6"
Oct 30 14:08:26.638221 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.634760    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw2l8\" (UniqueName: \"kubernetes.io/projected/e378bf44-e323-460f-9096-fd8835ad91b8-kube-api-access-zw2l8\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6\" (UID: \"e378bf44-e323-460f-9096-fd8835ad91b8\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6"
Oct 30 14:08:26.734227 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:26.734183    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6"
Oct 30 14:08:27.062525 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:27.062488    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6"]
Oct 30 14:08:27.073189 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:27.073145    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 14:08:27.073327 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:27.073245    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 14:08:27.073327 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:27.073290    5093 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 14:08:27.995812 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:27.995770    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6" event={"ID":"e378bf44-e323-460f-9096-fd8835ad91b8","Type":"ContainerStarted","Data":"04f1f3b38a412348f18d73b5e3ceb7bf41de1f7e199e437a04ea1aa735cd2f7b"}
Oct 30 14:08:27.996266 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:27.995824    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6" event={"ID":"e378bf44-e323-460f-9096-fd8835ad91b8","Type":"ContainerStarted","Data":"ea32d7e8dc3aa76870b2e51403179d26e415ca60d1fbb1fbf0eb3769c5966ad3"}
Oct 30 14:08:28.062155 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:28.061896    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6" podStartSLOduration=2.061877955 podStartE2EDuration="2.061877955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:08:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:08:28.060646486 +0000 UTC m=+3927.207913726" watchObservedRunningTime="2026-10-30 14:08:28.061877955 +0000 UTC m=+3927.209145195"
Oct 30 14:08:28.744151 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:28.743976    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6"
Oct 30 14:08:28.758153 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:28.757382    5093 patch_prober.go:29] interesting pod/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Oct 30 14:08:28.758153 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:28.757453    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6" podUID="e378bf44-e323-460f-9096-fd8835ad91b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Oct 30 14:08:29.740574 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:29.740530    5093 patch_prober.go:29] interesting pod/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Oct 30 14:08:29.741286 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:29.741247    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6" podUID="e378bf44-e323-460f-9096-fd8835ad91b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Oct 30 14:08:29.949225 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:29.949184    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-f12b80-5dbbcf49b-tqx8w"]
Oct 30 14:08:29.965143 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:29.965089    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f12b80-57dc694575-72wmr"]
Oct 30 14:08:30.738155 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:30.736552    5093 patch_prober.go:29] interesting pod/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Oct 30 14:08:30.738155 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:30.736648    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6" podUID="e378bf44-e323-460f-9096-fd8835ad91b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Oct 30 14:08:31.740157 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:31.739037    5093 patch_prober.go:29] interesting pod/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Oct 30 14:08:31.740157 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:31.739101    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6" podUID="e378bf44-e323-460f-9096-fd8835ad91b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Oct 30 14:08:32.736561 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:32.736520    5093 patch_prober.go:29] interesting pod/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Oct 30 14:08:32.736865 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:32.736836    5093 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6" podUID="e378bf44-e323-460f-9096-fd8835ad91b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Oct 30 14:08:33.740673 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:33.740635    5093 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6"
Oct 30 14:08:33.741699 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:33.741668    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6"
Oct 30 14:08:33.743334 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:33.743309    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6"
Oct 30 14:08:34.949817 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:34.949774    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f12b80-5dbbcf49b-tqx8w" podUID="4b8aed42-a18f-4cb3-aa5d-7a3cf0345391" containerName="payload-processing" containerID="cri-o://0e3f6d068a4edc1a11e2bf27491b93ab500201a6664122b68680d843db098aee" gracePeriod=25
Oct 30 14:08:34.966038 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:34.965962    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f12b80-57dc694575-72wmr" podUID="3a707991-287b-4944-a992-72e750b13a89" containerName="payload-pre-processing" containerID="cri-o://1b791ef6b49e5ee40648df4827c9844c9675b705f7548a30a51315dab3ddf6b7" gracePeriod=25
Oct 30 14:08:36.667752 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:36.667710    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-5852ff-6dcd476466-j2lxz"]
Oct 30 14:08:36.668185 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:36.667910    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5852ff-6dcd476466-j2lxz" podUID="a50e3626-0c42-47fc-83c3-a96c1889335c" containerName="maas-api" containerID="cri-o://fcb5c5d8a41ec77b4f2a254765da0a5f830d5bf575be354215f96cd781f2d9c9" gracePeriod=30
Oct 30 14:08:36.993205 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:36.993173    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5852ff-6dcd476466-j2lxz"
Oct 30 14:08:37.011783 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:37.011755    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a50e3626-0c42-47fc-83c3-a96c1889335c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a50e3626-0c42-47fc-83c3-a96c1889335c-openshift-service-ca\") pod \"a50e3626-0c42-47fc-83c3-a96c1889335c\" (UID: \"a50e3626-0c42-47fc-83c3-a96c1889335c\") "
Oct 30 14:08:37.011958 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:37.011843    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a50e3626-0c42-47fc-83c3-a96c1889335c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a50e3626-0c42-47fc-83c3-a96c1889335c-maas-api-tls\") pod \"a50e3626-0c42-47fc-83c3-a96c1889335c\" (UID: \"a50e3626-0c42-47fc-83c3-a96c1889335c\") "
Oct 30 14:08:37.012051 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:37.011999    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a50e3626-0c42-47fc-83c3-a96c1889335c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a50e3626-0c42-47fc-83c3-a96c1889335c-metrics-certs\") pod \"a50e3626-0c42-47fc-83c3-a96c1889335c\" (UID: \"a50e3626-0c42-47fc-83c3-a96c1889335c\") "
Oct 30 14:08:37.012116 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:37.012061    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a50e3626-0c42-47fc-83c3-a96c1889335c-kube-api-access-v2bq8\" (UniqueName: \"kubernetes.io/projected/a50e3626-0c42-47fc-83c3-a96c1889335c-kube-api-access-v2bq8\") pod \"a50e3626-0c42-47fc-83c3-a96c1889335c\" (UID: \"a50e3626-0c42-47fc-83c3-a96c1889335c\") "
Oct 30 14:08:37.012215 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:37.012154    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a50e3626-0c42-47fc-83c3-a96c1889335c-openshift-service-ca" pod "a50e3626-0c42-47fc-83c3-a96c1889335c" (UID: "a50e3626-0c42-47fc-83c3-a96c1889335c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:08:37.012684 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:37.012662    5093 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a50e3626-0c42-47fc-83c3-a96c1889335c-openshift-service-ca\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:08:37.014115 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:37.014090    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a50e3626-0c42-47fc-83c3-a96c1889335c-maas-api-tls" pod "a50e3626-0c42-47fc-83c3-a96c1889335c" (UID: "a50e3626-0c42-47fc-83c3-a96c1889335c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:08:37.014334 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:37.014308    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a50e3626-0c42-47fc-83c3-a96c1889335c-kube-api-access-v2bq8" pod "a50e3626-0c42-47fc-83c3-a96c1889335c" (UID: "a50e3626-0c42-47fc-83c3-a96c1889335c"). InnerVolumeSpecName "kube-api-access-v2bq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:08:37.014448 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:37.014428    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a50e3626-0c42-47fc-83c3-a96c1889335c-metrics-certs" pod "a50e3626-0c42-47fc-83c3-a96c1889335c" (UID: "a50e3626-0c42-47fc-83c3-a96c1889335c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:08:37.087578 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:37.087515    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f12b80-57dc694575-72wmr" podUID="3a707991-287b-4944-a992-72e750b13a89" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:08:37.098080 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:37.098046    5093 generic.go:356] "Generic (PLEG): container finished" podID="a50e3626-0c42-47fc-83c3-a96c1889335c" containerID="fcb5c5d8a41ec77b4f2a254765da0a5f830d5bf575be354215f96cd781f2d9c9" exitCode=0
Oct 30 14:08:37.098291 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:37.098103    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5852ff-6dcd476466-j2lxz"
Oct 30 14:08:37.098291 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:37.098158    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5852ff-6dcd476466-j2lxz" event={"ID":"a50e3626-0c42-47fc-83c3-a96c1889335c","Type":"ContainerDied","Data":"fcb5c5d8a41ec77b4f2a254765da0a5f830d5bf575be354215f96cd781f2d9c9"}
Oct 30 14:08:37.098291 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:37.098252    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5852ff-6dcd476466-j2lxz" event={"ID":"a50e3626-0c42-47fc-83c3-a96c1889335c","Type":"ContainerDied","Data":"f54e3df7c0ae5e6e3aa78ae96795ed4c9d4dea624a3416831ed58a31ac3aedcb"}
Oct 30 14:08:37.098291 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:37.098282    5093 scope.go:122] "RemoveContainer" containerID="fcb5c5d8a41ec77b4f2a254765da0a5f830d5bf575be354215f96cd781f2d9c9"
Oct 30 14:08:37.109651 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:37.109628    5093 scope.go:122] "RemoveContainer" containerID="fcb5c5d8a41ec77b4f2a254765da0a5f830d5bf575be354215f96cd781f2d9c9"
Oct 30 14:08:37.109939 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:08:37.109919    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcb5c5d8a41ec77b4f2a254765da0a5f830d5bf575be354215f96cd781f2d9c9\": container with ID starting with fcb5c5d8a41ec77b4f2a254765da0a5f830d5bf575be354215f96cd781f2d9c9 not found: ID does not exist" containerID="fcb5c5d8a41ec77b4f2a254765da0a5f830d5bf575be354215f96cd781f2d9c9"
Oct 30 14:08:37.109988 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:37.109951    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcb5c5d8a41ec77b4f2a254765da0a5f830d5bf575be354215f96cd781f2d9c9"} err="failed to get container status \"fcb5c5d8a41ec77b4f2a254765da0a5f830d5bf575be354215f96cd781f2d9c9\": rpc error: code = NotFound desc = could not find container \"fcb5c5d8a41ec77b4f2a254765da0a5f830d5bf575be354215f96cd781f2d9c9\": container with ID starting with fcb5c5d8a41ec77b4f2a254765da0a5f830d5bf575be354215f96cd781f2d9c9 not found: ID does not exist"
Oct 30 14:08:37.113846 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:37.113822    5093 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a50e3626-0c42-47fc-83c3-a96c1889335c-maas-api-tls\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:08:37.113846 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:37.113845    5093 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a50e3626-0c42-47fc-83c3-a96c1889335c-metrics-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:08:37.114013 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:37.113860    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2bq8\" (UniqueName: \"kubernetes.io/projected/a50e3626-0c42-47fc-83c3-a96c1889335c-kube-api-access-v2bq8\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:08:37.153517 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:37.153480    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-5852ff-6dcd476466-j2lxz"]
Oct 30 14:08:37.157676 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:37.157640    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-5852ff-6dcd476466-j2lxz"]
Oct 30 14:08:38.094149 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:38.094101    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f12b80-5dbbcf49b-tqx8w" podUID="4b8aed42-a18f-4cb3-aa5d-7a3cf0345391" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:08:39.002043 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:39.002008    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a50e3626-0c42-47fc-83c3-a96c1889335c" path="/var/lib/kubelet/pods/a50e3626-0c42-47fc-83c3-a96c1889335c/volumes"
Oct 30 14:08:47.087880 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:47.087834    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f12b80-57dc694575-72wmr" podUID="3a707991-287b-4944-a992-72e750b13a89" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:08:48.094294 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:48.094243    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f12b80-5dbbcf49b-tqx8w" podUID="4b8aed42-a18f-4cb3-aa5d-7a3cf0345391" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:08:51.276500 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:08:51.276420    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 14:08:55.252533 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:55.252492    5093 generic.go:356] "Generic (PLEG): container finished" podID="4b8aed42-a18f-4cb3-aa5d-7a3cf0345391" containerID="0e3f6d068a4edc1a11e2bf27491b93ab500201a6664122b68680d843db098aee" exitCode=0
Oct 30 14:08:55.252993 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:55.252743    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f12b80-5dbbcf49b-tqx8w" event={"ID":"4b8aed42-a18f-4cb3-aa5d-7a3cf0345391","Type":"ContainerDied","Data":"0e3f6d068a4edc1a11e2bf27491b93ab500201a6664122b68680d843db098aee"}
Oct 30 14:08:55.252993 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:55.252793    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f12b80-5dbbcf49b-tqx8w" event={"ID":"4b8aed42-a18f-4cb3-aa5d-7a3cf0345391","Type":"ContainerDied","Data":"98b7f41c2c9142dd3dae4f97f0e8b175b301021889f5f7e742cb8c0c58f4e079"}
Oct 30 14:08:55.252993 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:55.252815    5093 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98b7f41c2c9142dd3dae4f97f0e8b175b301021889f5f7e742cb8c0c58f4e079"
Oct 30 14:08:55.254780 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:55.254755    5093 generic.go:356] "Generic (PLEG): container finished" podID="3a707991-287b-4944-a992-72e750b13a89" containerID="1b791ef6b49e5ee40648df4827c9844c9675b705f7548a30a51315dab3ddf6b7" exitCode=0
Oct 30 14:08:55.254934 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:55.254805    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f12b80-57dc694575-72wmr" event={"ID":"3a707991-287b-4944-a992-72e750b13a89","Type":"ContainerDied","Data":"1b791ef6b49e5ee40648df4827c9844c9675b705f7548a30a51315dab3ddf6b7"}
Oct 30 14:08:55.273790 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:55.273759    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f12b80-5dbbcf49b-tqx8w"
Oct 30 14:08:55.303158 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:55.303118    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f12b80-57dc694575-72wmr"
Oct 30 14:08:55.384957 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:55.384922    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b8aed42-a18f-4cb3-aa5d-7a3cf0345391-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b8aed42-a18f-4cb3-aa5d-7a3cf0345391-plugins-config-volume\") pod \"4b8aed42-a18f-4cb3-aa5d-7a3cf0345391\" (UID: \"4b8aed42-a18f-4cb3-aa5d-7a3cf0345391\") "
Oct 30 14:08:55.385250 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:55.385229    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b8aed42-a18f-4cb3-aa5d-7a3cf0345391-kube-api-access-kznq6\" (UniqueName: \"kubernetes.io/projected/4b8aed42-a18f-4cb3-aa5d-7a3cf0345391-kube-api-access-kznq6\") pod \"4b8aed42-a18f-4cb3-aa5d-7a3cf0345391\" (UID: \"4b8aed42-a18f-4cb3-aa5d-7a3cf0345391\") "
Oct 30 14:08:55.385408 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:55.385383    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b8aed42-a18f-4cb3-aa5d-7a3cf0345391-plugins-config-volume" pod "4b8aed42-a18f-4cb3-aa5d-7a3cf0345391" (UID: "4b8aed42-a18f-4cb3-aa5d-7a3cf0345391"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:08:55.385695 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:55.385677    5093 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b8aed42-a18f-4cb3-aa5d-7a3cf0345391-plugins-config-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:08:55.387454 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:55.387428    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b8aed42-a18f-4cb3-aa5d-7a3cf0345391-kube-api-access-kznq6" pod "4b8aed42-a18f-4cb3-aa5d-7a3cf0345391" (UID: "4b8aed42-a18f-4cb3-aa5d-7a3cf0345391"). InnerVolumeSpecName "kube-api-access-kznq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:08:55.486613 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:55.486573    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a707991-287b-4944-a992-72e750b13a89-kube-api-access-lv89t\" (UniqueName: \"kubernetes.io/projected/3a707991-287b-4944-a992-72e750b13a89-kube-api-access-lv89t\") pod \"3a707991-287b-4944-a992-72e750b13a89\" (UID: \"3a707991-287b-4944-a992-72e750b13a89\") "
Oct 30 14:08:55.486823 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:55.486808    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a707991-287b-4944-a992-72e750b13a89-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a707991-287b-4944-a992-72e750b13a89-plugins-config-volume\") pod \"3a707991-287b-4944-a992-72e750b13a89\" (UID: \"3a707991-287b-4944-a992-72e750b13a89\") "
Oct 30 14:08:55.487320 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:55.487287    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a707991-287b-4944-a992-72e750b13a89-plugins-config-volume" pod "3a707991-287b-4944-a992-72e750b13a89" (UID: "3a707991-287b-4944-a992-72e750b13a89"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:08:55.487320 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:55.487305    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kznq6\" (UniqueName: \"kubernetes.io/projected/4b8aed42-a18f-4cb3-aa5d-7a3cf0345391-kube-api-access-kznq6\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:08:55.488892 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:55.488866    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a707991-287b-4944-a992-72e750b13a89-kube-api-access-lv89t" pod "3a707991-287b-4944-a992-72e750b13a89" (UID: "3a707991-287b-4944-a992-72e750b13a89"). InnerVolumeSpecName "kube-api-access-lv89t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:08:55.588656 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:55.588564    5093 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a707991-287b-4944-a992-72e750b13a89-plugins-config-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:08:55.588656 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:55.588600    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv89t\" (UniqueName: \"kubernetes.io/projected/3a707991-287b-4944-a992-72e750b13a89-kube-api-access-lv89t\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:08:56.263436 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:56.263343    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f12b80-57dc694575-72wmr"
Oct 30 14:08:56.263436 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:56.263375    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f12b80-57dc694575-72wmr" event={"ID":"3a707991-287b-4944-a992-72e750b13a89","Type":"ContainerDied","Data":"cd458aee00e90cdef204c5c5d458bc04b5df11f6764b5b865954442e42675049"}
Oct 30 14:08:56.263987 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:56.263444    5093 scope.go:122] "RemoveContainer" containerID="1b791ef6b49e5ee40648df4827c9844c9675b705f7548a30a51315dab3ddf6b7"
Oct 30 14:08:56.263987 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:56.263679    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f12b80-5dbbcf49b-tqx8w"
Oct 30 14:08:56.318280 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:56.318243    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-f12b80-5dbbcf49b-tqx8w"]
Oct 30 14:08:56.331058 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:56.331021    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-f12b80-5dbbcf49b-tqx8w"]
Oct 30 14:08:56.358446 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:56.358407    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f12b80-57dc694575-72wmr"]
Oct 30 14:08:56.361784 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:56.361750    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f12b80-57dc694575-72wmr"]
Oct 30 14:08:56.550402 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:56.550316    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f12b80-6c694985fd-g8lxp"]
Oct 30 14:08:56.550659 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:56.550611    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f12b80-6c694985fd-g8lxp" podUID="0d63ba90-2e39-44a9-aa12-ec1adf2dc23f" containerName="maas-api" containerID="cri-o://8d753f050bd7996a9aec9622c4b5309ced94f75bcb91015edac29b2baf598214" gracePeriod=30
Oct 30 14:08:56.861915 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:56.861873    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f12b80-6c694985fd-g8lxp"
Oct 30 14:08:57.001755 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:57.001714    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d63ba90-2e39-44a9-aa12-ec1adf2dc23f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d63ba90-2e39-44a9-aa12-ec1adf2dc23f-metrics-certs\") pod \"0d63ba90-2e39-44a9-aa12-ec1adf2dc23f\" (UID: \"0d63ba90-2e39-44a9-aa12-ec1adf2dc23f\") "
Oct 30 14:08:57.001973 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:57.001774    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d63ba90-2e39-44a9-aa12-ec1adf2dc23f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d63ba90-2e39-44a9-aa12-ec1adf2dc23f-openshift-service-ca\") pod \"0d63ba90-2e39-44a9-aa12-ec1adf2dc23f\" (UID: \"0d63ba90-2e39-44a9-aa12-ec1adf2dc23f\") "
Oct 30 14:08:57.001973 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:57.001840    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d63ba90-2e39-44a9-aa12-ec1adf2dc23f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d63ba90-2e39-44a9-aa12-ec1adf2dc23f-maas-api-tls\") pod \"0d63ba90-2e39-44a9-aa12-ec1adf2dc23f\" (UID: \"0d63ba90-2e39-44a9-aa12-ec1adf2dc23f\") "
Oct 30 14:08:57.001973 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:57.001896    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a707991-287b-4944-a992-72e750b13a89" path="/var/lib/kubelet/pods/3a707991-287b-4944-a992-72e750b13a89/volumes"
Oct 30 14:08:57.002203 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:57.001904    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d63ba90-2e39-44a9-aa12-ec1adf2dc23f-kube-api-access-mzd7r\" (UniqueName: \"kubernetes.io/projected/0d63ba90-2e39-44a9-aa12-ec1adf2dc23f-kube-api-access-mzd7r\") pod \"0d63ba90-2e39-44a9-aa12-ec1adf2dc23f\" (UID: \"0d63ba90-2e39-44a9-aa12-ec1adf2dc23f\") "
Oct 30 14:08:57.002267 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:57.002211    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d63ba90-2e39-44a9-aa12-ec1adf2dc23f-openshift-service-ca" pod "0d63ba90-2e39-44a9-aa12-ec1adf2dc23f" (UID: "0d63ba90-2e39-44a9-aa12-ec1adf2dc23f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:08:57.002603 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:57.002575    5093 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d63ba90-2e39-44a9-aa12-ec1adf2dc23f-openshift-service-ca\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:08:57.002723 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:57.002649    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b8aed42-a18f-4cb3-aa5d-7a3cf0345391" path="/var/lib/kubelet/pods/4b8aed42-a18f-4cb3-aa5d-7a3cf0345391/volumes"
Oct 30 14:08:57.004231 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:57.004205    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d63ba90-2e39-44a9-aa12-ec1adf2dc23f-metrics-certs" pod "0d63ba90-2e39-44a9-aa12-ec1adf2dc23f" (UID: "0d63ba90-2e39-44a9-aa12-ec1adf2dc23f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:08:57.004385 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:57.004368    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d63ba90-2e39-44a9-aa12-ec1adf2dc23f-maas-api-tls" pod "0d63ba90-2e39-44a9-aa12-ec1adf2dc23f" (UID: "0d63ba90-2e39-44a9-aa12-ec1adf2dc23f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:08:57.004490 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:57.004465    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d63ba90-2e39-44a9-aa12-ec1adf2dc23f-kube-api-access-mzd7r" pod "0d63ba90-2e39-44a9-aa12-ec1adf2dc23f" (UID: "0d63ba90-2e39-44a9-aa12-ec1adf2dc23f"). InnerVolumeSpecName "kube-api-access-mzd7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:08:57.105224 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:57.104662    5093 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d63ba90-2e39-44a9-aa12-ec1adf2dc23f-metrics-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:08:57.105224 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:57.104725    5093 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d63ba90-2e39-44a9-aa12-ec1adf2dc23f-maas-api-tls\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:08:57.105224 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:57.105055    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzd7r\" (UniqueName: \"kubernetes.io/projected/0d63ba90-2e39-44a9-aa12-ec1adf2dc23f-kube-api-access-mzd7r\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:08:57.273211 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:57.273113    5093 generic.go:356] "Generic (PLEG): container finished" podID="0d63ba90-2e39-44a9-aa12-ec1adf2dc23f" containerID="8d753f050bd7996a9aec9622c4b5309ced94f75bcb91015edac29b2baf598214" exitCode=0
Oct 30 14:08:57.273211 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:57.273198    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f12b80-6c694985fd-g8lxp" event={"ID":"0d63ba90-2e39-44a9-aa12-ec1adf2dc23f","Type":"ContainerDied","Data":"8d753f050bd7996a9aec9622c4b5309ced94f75bcb91015edac29b2baf598214"}
Oct 30 14:08:57.273736 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:57.273247    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f12b80-6c694985fd-g8lxp" event={"ID":"0d63ba90-2e39-44a9-aa12-ec1adf2dc23f","Type":"ContainerDied","Data":"bf89ec92b2b64e4469eca6f944ac1ff1f7de40540d19ced6e2c50c5f0d8f3b6d"}
Oct 30 14:08:57.273736 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:57.273176    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f12b80-6c694985fd-g8lxp"
Oct 30 14:08:57.273736 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:57.273277    5093 scope.go:122] "RemoveContainer" containerID="8d753f050bd7996a9aec9622c4b5309ced94f75bcb91015edac29b2baf598214"
Oct 30 14:08:57.284495 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:57.284471    5093 scope.go:122] "RemoveContainer" containerID="8d753f050bd7996a9aec9622c4b5309ced94f75bcb91015edac29b2baf598214"
Oct 30 14:08:57.284804 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:08:57.284785    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d753f050bd7996a9aec9622c4b5309ced94f75bcb91015edac29b2baf598214\": container with ID starting with 8d753f050bd7996a9aec9622c4b5309ced94f75bcb91015edac29b2baf598214 not found: ID does not exist" containerID="8d753f050bd7996a9aec9622c4b5309ced94f75bcb91015edac29b2baf598214"
Oct 30 14:08:57.284850 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:57.284818    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d753f050bd7996a9aec9622c4b5309ced94f75bcb91015edac29b2baf598214"} err="failed to get container status \"8d753f050bd7996a9aec9622c4b5309ced94f75bcb91015edac29b2baf598214\": rpc error: code = NotFound desc = could not find container \"8d753f050bd7996a9aec9622c4b5309ced94f75bcb91015edac29b2baf598214\": container with ID starting with 8d753f050bd7996a9aec9622c4b5309ced94f75bcb91015edac29b2baf598214 not found: ID does not exist"
Oct 30 14:08:57.329616 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:57.329575    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f12b80-6c694985fd-g8lxp"]
Oct 30 14:08:57.335317 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:57.335275    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f12b80-6c694985fd-g8lxp"]
Oct 30 14:08:59.005172 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:08:59.005121    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d63ba90-2e39-44a9-aa12-ec1adf2dc23f" path="/var/lib/kubelet/pods/0d63ba90-2e39-44a9-aa12-ec1adf2dc23f/volumes"
Oct 30 14:09:03.093891 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:03.093859    5093 scope.go:122] "RemoveContainer" containerID="0e3f6d068a4edc1a11e2bf27491b93ab500201a6664122b68680d843db098aee"
Oct 30 14:09:10.164263 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:10.164220    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb"]
Oct 30 14:09:10.177065 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:10.177028    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv"]
Oct 30 14:09:10.183526 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:10.183493    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx"]
Oct 30 14:09:21.170055 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:09:21.170001    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 14:09:25.187970 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.187914    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb" podUID="c4b4e00e-f58d-4a4e-b7db-19f724360db1" containerName="main" containerID="cri-o://b2a5c79681857ff27077fb57511fd57f01046ba1192945042b9122a373391b3e" gracePeriod=45
Oct 30 14:09:25.202310 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.202266    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv" podUID="d6a44981-104f-4d3e-9937-d184a6ddc482" containerName="main" containerID="cri-o://916ddc84c1bb68343ff23094fa39c91d7f97250e51f41a27d30895d605577072" gracePeriod=45
Oct 30 14:09:25.205486 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.205348    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx" podUID="ba156eeb-9955-4f9a-857b-2c4d502a5fcc" containerName="main" containerID="cri-o://015fe89168d85d6179e16b6f6b6af7cd10529d8d19a277dc8719c3727ff83efa" gracePeriod=45
Oct 30 14:09:25.483162 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.483061    5093 generic.go:356] "Generic (PLEG): container finished" podID="ba156eeb-9955-4f9a-857b-2c4d502a5fcc" containerID="015fe89168d85d6179e16b6f6b6af7cd10529d8d19a277dc8719c3727ff83efa" exitCode=0
Oct 30 14:09:25.483369 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.483219    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx" event={"ID":"ba156eeb-9955-4f9a-857b-2c4d502a5fcc","Type":"ContainerDied","Data":"015fe89168d85d6179e16b6f6b6af7cd10529d8d19a277dc8719c3727ff83efa"}
Oct 30 14:09:25.485659 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.485626    5093 generic.go:356] "Generic (PLEG): container finished" podID="c4b4e00e-f58d-4a4e-b7db-19f724360db1" containerID="b2a5c79681857ff27077fb57511fd57f01046ba1192945042b9122a373391b3e" exitCode=0
Oct 30 14:09:25.485819 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.485769    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb" event={"ID":"c4b4e00e-f58d-4a4e-b7db-19f724360db1","Type":"ContainerDied","Data":"b2a5c79681857ff27077fb57511fd57f01046ba1192945042b9122a373391b3e"}
Oct 30 14:09:25.488846 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.488815    5093 generic.go:356] "Generic (PLEG): container finished" podID="d6a44981-104f-4d3e-9937-d184a6ddc482" containerID="916ddc84c1bb68343ff23094fa39c91d7f97250e51f41a27d30895d605577072" exitCode=0
Oct 30 14:09:25.489005 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.488861    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv" event={"ID":"d6a44981-104f-4d3e-9937-d184a6ddc482","Type":"ContainerDied","Data":"916ddc84c1bb68343ff23094fa39c91d7f97250e51f41a27d30895d605577072"}
Oct 30 14:09:25.551201 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.551177    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv"
Oct 30 14:09:25.553207 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.553177    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb"
Oct 30 14:09:25.575332 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.575295    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx"
Oct 30 14:09:25.589530 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.589497    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4b4e00e-f58d-4a4e-b7db-19f724360db1-kube-api-access-tcgxf\" (UniqueName: \"kubernetes.io/projected/c4b4e00e-f58d-4a4e-b7db-19f724360db1-kube-api-access-tcgxf\") pod \"c4b4e00e-f58d-4a4e-b7db-19f724360db1\" (UID: \"c4b4e00e-f58d-4a4e-b7db-19f724360db1\") "
Oct 30 14:09:25.589700 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.589546    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4b4e00e-f58d-4a4e-b7db-19f724360db1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4b4e00e-f58d-4a4e-b7db-19f724360db1-tmp-dir\") pod \"c4b4e00e-f58d-4a4e-b7db-19f724360db1\" (UID: \"c4b4e00e-f58d-4a4e-b7db-19f724360db1\") "
Oct 30 14:09:25.589700 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.589585    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6a44981-104f-4d3e-9937-d184a6ddc482-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6a44981-104f-4d3e-9937-d184a6ddc482-tmp-dir\") pod \"d6a44981-104f-4d3e-9937-d184a6ddc482\" (UID: \"d6a44981-104f-4d3e-9937-d184a6ddc482\") "
Oct 30 14:09:25.589700 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.589604    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4b4e00e-f58d-4a4e-b7db-19f724360db1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4b4e00e-f58d-4a4e-b7db-19f724360db1-model-cache\") pod \"c4b4e00e-f58d-4a4e-b7db-19f724360db1\" (UID: \"c4b4e00e-f58d-4a4e-b7db-19f724360db1\") "
Oct 30 14:09:25.589700 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.589686    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6a44981-104f-4d3e-9937-d184a6ddc482-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6a44981-104f-4d3e-9937-d184a6ddc482-model-cache\") pod \"d6a44981-104f-4d3e-9937-d184a6ddc482\" (UID: \"d6a44981-104f-4d3e-9937-d184a6ddc482\") "
Oct 30 14:09:25.589842 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.589719    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6a44981-104f-4d3e-9937-d184a6ddc482-tls-certs\" (UniqueName: \"kubernetes.io/secret/d6a44981-104f-4d3e-9937-d184a6ddc482-tls-certs\") pod \"d6a44981-104f-4d3e-9937-d184a6ddc482\" (UID: \"d6a44981-104f-4d3e-9937-d184a6ddc482\") "
Oct 30 14:09:25.589842 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.589779    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6a44981-104f-4d3e-9937-d184a6ddc482-kube-api-access-lh6h7\" (UniqueName: \"kubernetes.io/projected/d6a44981-104f-4d3e-9937-d184a6ddc482-kube-api-access-lh6h7\") pod \"d6a44981-104f-4d3e-9937-d184a6ddc482\" (UID: \"d6a44981-104f-4d3e-9937-d184a6ddc482\") "
Oct 30 14:09:25.589842 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.589805    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6a44981-104f-4d3e-9937-d184a6ddc482-home\" (UniqueName: \"kubernetes.io/empty-dir/d6a44981-104f-4d3e-9937-d184a6ddc482-home\") pod \"d6a44981-104f-4d3e-9937-d184a6ddc482\" (UID: \"d6a44981-104f-4d3e-9937-d184a6ddc482\") "
Oct 30 14:09:25.589938 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.589856    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6a44981-104f-4d3e-9937-d184a6ddc482-tmp-dir" pod "d6a44981-104f-4d3e-9937-d184a6ddc482" (UID: "d6a44981-104f-4d3e-9937-d184a6ddc482"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:09:25.589938 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.589872    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6a44981-104f-4d3e-9937-d184a6ddc482-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6a44981-104f-4d3e-9937-d184a6ddc482-dshm\") pod \"d6a44981-104f-4d3e-9937-d184a6ddc482\" (UID: \"d6a44981-104f-4d3e-9937-d184a6ddc482\") "
Oct 30 14:09:25.590072 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.589946    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4b4e00e-f58d-4a4e-b7db-19f724360db1-tls-certs\" (UniqueName: \"kubernetes.io/secret/c4b4e00e-f58d-4a4e-b7db-19f724360db1-tls-certs\") pod \"c4b4e00e-f58d-4a4e-b7db-19f724360db1\" (UID: \"c4b4e00e-f58d-4a4e-b7db-19f724360db1\") "
Oct 30 14:09:25.590072 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.589978    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4b4e00e-f58d-4a4e-b7db-19f724360db1-home\" (UniqueName: \"kubernetes.io/empty-dir/c4b4e00e-f58d-4a4e-b7db-19f724360db1-home\") pod \"c4b4e00e-f58d-4a4e-b7db-19f724360db1\" (UID: \"c4b4e00e-f58d-4a4e-b7db-19f724360db1\") "
Oct 30 14:09:25.590072 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.590009    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6a44981-104f-4d3e-9937-d184a6ddc482-model-cache" pod "d6a44981-104f-4d3e-9937-d184a6ddc482" (UID: "d6a44981-104f-4d3e-9937-d184a6ddc482"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:09:25.590267 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.590108    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4b4e00e-f58d-4a4e-b7db-19f724360db1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4b4e00e-f58d-4a4e-b7db-19f724360db1-dshm\") pod \"c4b4e00e-f58d-4a4e-b7db-19f724360db1\" (UID: \"c4b4e00e-f58d-4a4e-b7db-19f724360db1\") "
Oct 30 14:09:25.590267 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.590165    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4b4e00e-f58d-4a4e-b7db-19f724360db1-model-cache" pod "c4b4e00e-f58d-4a4e-b7db-19f724360db1" (UID: "c4b4e00e-f58d-4a4e-b7db-19f724360db1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:09:25.590518 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.590490    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6a44981-104f-4d3e-9937-d184a6ddc482-home" pod "d6a44981-104f-4d3e-9937-d184a6ddc482" (UID: "d6a44981-104f-4d3e-9937-d184a6ddc482"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:09:25.590607 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.590561    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4b4e00e-f58d-4a4e-b7db-19f724360db1-tmp-dir" pod "c4b4e00e-f58d-4a4e-b7db-19f724360db1" (UID: "c4b4e00e-f58d-4a4e-b7db-19f724360db1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:09:25.590784 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.590760    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4b4e00e-f58d-4a4e-b7db-19f724360db1-home" pod "c4b4e00e-f58d-4a4e-b7db-19f724360db1" (UID: "c4b4e00e-f58d-4a4e-b7db-19f724360db1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:09:25.590939 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.590919    5093 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4b4e00e-f58d-4a4e-b7db-19f724360db1-tmp-dir\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:09:25.591014 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.590943    5093 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6a44981-104f-4d3e-9937-d184a6ddc482-tmp-dir\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:09:25.591014 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.590957    5093 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4b4e00e-f58d-4a4e-b7db-19f724360db1-model-cache\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:09:25.591014 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.590970    5093 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6a44981-104f-4d3e-9937-d184a6ddc482-model-cache\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:09:25.591014 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.590981    5093 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6a44981-104f-4d3e-9937-d184a6ddc482-home\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:09:25.591014 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.590992    5093 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4b4e00e-f58d-4a4e-b7db-19f724360db1-home\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:09:25.592958 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.592925    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6a44981-104f-4d3e-9937-d184a6ddc482-dshm" pod "d6a44981-104f-4d3e-9937-d184a6ddc482" (UID: "d6a44981-104f-4d3e-9937-d184a6ddc482"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:09:25.593124 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.592887    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6a44981-104f-4d3e-9937-d184a6ddc482-tls-certs" pod "d6a44981-104f-4d3e-9937-d184a6ddc482" (UID: "d6a44981-104f-4d3e-9937-d184a6ddc482"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:09:25.593124 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.593035    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4b4e00e-f58d-4a4e-b7db-19f724360db1-kube-api-access-tcgxf" pod "c4b4e00e-f58d-4a4e-b7db-19f724360db1" (UID: "c4b4e00e-f58d-4a4e-b7db-19f724360db1"). InnerVolumeSpecName "kube-api-access-tcgxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:09:25.593124 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.593067    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6a44981-104f-4d3e-9937-d184a6ddc482-kube-api-access-lh6h7" pod "d6a44981-104f-4d3e-9937-d184a6ddc482" (UID: "d6a44981-104f-4d3e-9937-d184a6ddc482"). InnerVolumeSpecName "kube-api-access-lh6h7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:09:25.593415 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.593296    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4b4e00e-f58d-4a4e-b7db-19f724360db1-tls-certs" pod "c4b4e00e-f58d-4a4e-b7db-19f724360db1" (UID: "c4b4e00e-f58d-4a4e-b7db-19f724360db1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:09:25.593951 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.593931    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4b4e00e-f58d-4a4e-b7db-19f724360db1-dshm" pod "c4b4e00e-f58d-4a4e-b7db-19f724360db1" (UID: "c4b4e00e-f58d-4a4e-b7db-19f724360db1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:09:25.692240 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.692197    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba156eeb-9955-4f9a-857b-2c4d502a5fcc-home\" (UniqueName: \"kubernetes.io/empty-dir/ba156eeb-9955-4f9a-857b-2c4d502a5fcc-home\") pod \"ba156eeb-9955-4f9a-857b-2c4d502a5fcc\" (UID: \"ba156eeb-9955-4f9a-857b-2c4d502a5fcc\") "
Oct 30 14:09:25.692576 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.692553    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba156eeb-9955-4f9a-857b-2c4d502a5fcc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba156eeb-9955-4f9a-857b-2c4d502a5fcc-tmp-dir\") pod \"ba156eeb-9955-4f9a-857b-2c4d502a5fcc\" (UID: \"ba156eeb-9955-4f9a-857b-2c4d502a5fcc\") "
Oct 30 14:09:25.692751 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.692729    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba156eeb-9955-4f9a-857b-2c4d502a5fcc-kube-api-access-f469c\" (UniqueName: \"kubernetes.io/projected/ba156eeb-9955-4f9a-857b-2c4d502a5fcc-kube-api-access-f469c\") pod \"ba156eeb-9955-4f9a-857b-2c4d502a5fcc\" (UID: \"ba156eeb-9955-4f9a-857b-2c4d502a5fcc\") "
Oct 30 14:09:25.692972 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.692956    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba156eeb-9955-4f9a-857b-2c4d502a5fcc-tls-certs\" (UniqueName: \"kubernetes.io/secret/ba156eeb-9955-4f9a-857b-2c4d502a5fcc-tls-certs\") pod \"ba156eeb-9955-4f9a-857b-2c4d502a5fcc\" (UID: \"ba156eeb-9955-4f9a-857b-2c4d502a5fcc\") "
Oct 30 14:09:25.693175 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.693114    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba156eeb-9955-4f9a-857b-2c4d502a5fcc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba156eeb-9955-4f9a-857b-2c4d502a5fcc-dshm\") pod \"ba156eeb-9955-4f9a-857b-2c4d502a5fcc\" (UID: \"ba156eeb-9955-4f9a-857b-2c4d502a5fcc\") "
Oct 30 14:09:25.693303 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.693288    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba156eeb-9955-4f9a-857b-2c4d502a5fcc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba156eeb-9955-4f9a-857b-2c4d502a5fcc-model-cache\") pod \"ba156eeb-9955-4f9a-857b-2c4d502a5fcc\" (UID: \"ba156eeb-9955-4f9a-857b-2c4d502a5fcc\") "
Oct 30 14:09:25.694055 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.694022    5093 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4b4e00e-f58d-4a4e-b7db-19f724360db1-dshm\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:09:25.694296 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.694278    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcgxf\" (UniqueName: \"kubernetes.io/projected/c4b4e00e-f58d-4a4e-b7db-19f724360db1-kube-api-access-tcgxf\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:09:25.694408 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.694396    5093 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6a44981-104f-4d3e-9937-d184a6ddc482-tls-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:09:25.694495 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.694485    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh6h7\" (UniqueName: \"kubernetes.io/projected/d6a44981-104f-4d3e-9937-d184a6ddc482-kube-api-access-lh6h7\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:09:25.694582 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.694571    5093 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6a44981-104f-4d3e-9937-d184a6ddc482-dshm\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:09:25.694659 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.694649    5093 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4b4e00e-f58d-4a4e-b7db-19f724360db1-tls-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:09:25.702772 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.692487    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba156eeb-9955-4f9a-857b-2c4d502a5fcc-home" pod "ba156eeb-9955-4f9a-857b-2c4d502a5fcc" (UID: "ba156eeb-9955-4f9a-857b-2c4d502a5fcc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:09:25.702772 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.694920    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba156eeb-9955-4f9a-857b-2c4d502a5fcc-model-cache" pod "ba156eeb-9955-4f9a-857b-2c4d502a5fcc" (UID: "ba156eeb-9955-4f9a-857b-2c4d502a5fcc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:09:25.702772 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.695972    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba156eeb-9955-4f9a-857b-2c4d502a5fcc-tmp-dir" pod "ba156eeb-9955-4f9a-857b-2c4d502a5fcc" (UID: "ba156eeb-9955-4f9a-857b-2c4d502a5fcc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:09:25.706099 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.706063    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba156eeb-9955-4f9a-857b-2c4d502a5fcc-tls-certs" pod "ba156eeb-9955-4f9a-857b-2c4d502a5fcc" (UID: "ba156eeb-9955-4f9a-857b-2c4d502a5fcc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:09:25.709644 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.709608    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba156eeb-9955-4f9a-857b-2c4d502a5fcc-dshm" pod "ba156eeb-9955-4f9a-857b-2c4d502a5fcc" (UID: "ba156eeb-9955-4f9a-857b-2c4d502a5fcc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:09:25.709781 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.709623    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba156eeb-9955-4f9a-857b-2c4d502a5fcc-kube-api-access-f469c" pod "ba156eeb-9955-4f9a-857b-2c4d502a5fcc" (UID: "ba156eeb-9955-4f9a-857b-2c4d502a5fcc"). InnerVolumeSpecName "kube-api-access-f469c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:09:25.797174 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.797047    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f469c\" (UniqueName: \"kubernetes.io/projected/ba156eeb-9955-4f9a-857b-2c4d502a5fcc-kube-api-access-f469c\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:09:25.797174 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.797091    5093 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba156eeb-9955-4f9a-857b-2c4d502a5fcc-tls-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:09:25.797174 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.797107    5093 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba156eeb-9955-4f9a-857b-2c4d502a5fcc-dshm\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:09:25.797174 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.797122    5093 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba156eeb-9955-4f9a-857b-2c4d502a5fcc-model-cache\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:09:25.797174 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.797148    5093 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba156eeb-9955-4f9a-857b-2c4d502a5fcc-home\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:09:25.797174 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:25.797159    5093 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba156eeb-9955-4f9a-857b-2c4d502a5fcc-tmp-dir\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:09:26.401595 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:26.401542    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv" podUID="d6a44981-104f-4d3e-9937-d184a6ddc482" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:09:26.496794 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:26.496697    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv"
Oct 30 14:09:26.496794 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:26.496727    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv" event={"ID":"d6a44981-104f-4d3e-9937-d184a6ddc482","Type":"ContainerDied","Data":"8878e62d4d3dd14c2621683747e7cd2af1be168f51623b5bf50d1ce073d211bc"}
Oct 30 14:09:26.496794 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:26.496789    5093 scope.go:122] "RemoveContainer" containerID="916ddc84c1bb68343ff23094fa39c91d7f97250e51f41a27d30895d605577072"
Oct 30 14:09:26.498893 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:26.498871    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx"
Oct 30 14:09:26.499026 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:26.498952    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx" event={"ID":"ba156eeb-9955-4f9a-857b-2c4d502a5fcc","Type":"ContainerDied","Data":"2e50c8bbb0d34c7d5706484c10e3d7885298114e766aabf4975929301e9032c4"}
Oct 30 14:09:26.500896 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:26.500872    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb"
Oct 30 14:09:26.501047 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:26.500891    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb" event={"ID":"c4b4e00e-f58d-4a4e-b7db-19f724360db1","Type":"ContainerDied","Data":"abcd0c618bfc497c28340c151fd4f319158ff79ac00917bafb06d5081257d4fd"}
Oct 30 14:09:26.511090 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:26.511051    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb" podUID="c4b4e00e-f58d-4a4e-b7db-19f724360db1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Oct 30 14:09:26.514093 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:26.513945    5093 scope.go:122] "RemoveContainer" containerID="015fe89168d85d6179e16b6f6b6af7cd10529d8d19a277dc8719c3727ff83efa"
Oct 30 14:09:26.525891 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:26.525858    5093 scope.go:122] "RemoveContainer" containerID="b2a5c79681857ff27077fb57511fd57f01046ba1192945042b9122a373391b3e"
Oct 30 14:09:26.557465 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:26.557419    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv"]
Oct 30 14:09:26.562553 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:26.562521    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-9bf8ad/premium-w4-9bf8ad-kserve-86c698998b-r2jbv"]
Oct 30 14:09:26.595730 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:26.595691    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb"]
Oct 30 14:09:26.602910 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:26.602864    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-9bf8ad/distinct-w4-9bf8ad-kserve-66d8c4fbdf-gz7sb"]
Oct 30 14:09:26.643353 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:26.643303    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx"]
Oct 30 14:09:26.650218 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:26.650168    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-9bf8ad/simulator-w4-9bf8ad-kserve-6589f99f6c-pvckx"]
Oct 30 14:09:26.999439 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:26.999395    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba156eeb-9955-4f9a-857b-2c4d502a5fcc" path="/var/lib/kubelet/pods/ba156eeb-9955-4f9a-857b-2c4d502a5fcc/volumes"
Oct 30 14:09:26.999887 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:26.999871    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4b4e00e-f58d-4a4e-b7db-19f724360db1" path="/var/lib/kubelet/pods/c4b4e00e-f58d-4a4e-b7db-19f724360db1/volumes"
Oct 30 14:09:27.000379 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:27.000354    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6a44981-104f-4d3e-9937-d184a6ddc482" path="/var/lib/kubelet/pods/d6a44981-104f-4d3e-9937-d184a6ddc482/volumes"
Oct 30 14:09:51.168834 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:09:51.168738    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 14:09:53.787717 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:53.787684    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-9bf8ad-6d48f89c8c-wzj8b"]
Oct 30 14:09:53.788256 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:53.787933    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9bf8ad-6d48f89c8c-wzj8b" podUID="75e20d5d-abc6-4682-8b47-0bb3abf6a790" containerName="maas-api" containerID="cri-o://d92274fdca9a94d2ec274c8f78e377b9d7b83c0ba5b97a70f1251764469cd9dd" gracePeriod=30
Oct 30 14:09:53.834257 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:53.834196    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9bf8ad-6d48f89c8c-wzj8b" podUID="75e20d5d-abc6-4682-8b47-0bb3abf6a790" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.73:8443/health\": dial tcp 10.132.0.73:8443: connect: connection refused"
Oct 30 14:09:54.103412 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:54.103381    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9bf8ad-6d48f89c8c-wzj8b"
Oct 30 14:09:54.182399 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:54.182344    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75e20d5d-abc6-4682-8b47-0bb3abf6a790-metrics-certs\" (UniqueName: \"kubernetes.io/secret/75e20d5d-abc6-4682-8b47-0bb3abf6a790-metrics-certs\") pod \"75e20d5d-abc6-4682-8b47-0bb3abf6a790\" (UID: \"75e20d5d-abc6-4682-8b47-0bb3abf6a790\") "
Oct 30 14:09:54.182629 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:54.182434    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75e20d5d-abc6-4682-8b47-0bb3abf6a790-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75e20d5d-abc6-4682-8b47-0bb3abf6a790-openshift-service-ca\") pod \"75e20d5d-abc6-4682-8b47-0bb3abf6a790\" (UID: \"75e20d5d-abc6-4682-8b47-0bb3abf6a790\") "
Oct 30 14:09:54.182629 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:54.182582    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75e20d5d-abc6-4682-8b47-0bb3abf6a790-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75e20d5d-abc6-4682-8b47-0bb3abf6a790-maas-api-tls\") pod \"75e20d5d-abc6-4682-8b47-0bb3abf6a790\" (UID: \"75e20d5d-abc6-4682-8b47-0bb3abf6a790\") "
Oct 30 14:09:54.182793 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:54.182633    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75e20d5d-abc6-4682-8b47-0bb3abf6a790-kube-api-access-4j2mm\" (UniqueName: \"kubernetes.io/projected/75e20d5d-abc6-4682-8b47-0bb3abf6a790-kube-api-access-4j2mm\") pod \"75e20d5d-abc6-4682-8b47-0bb3abf6a790\" (UID: \"75e20d5d-abc6-4682-8b47-0bb3abf6a790\") "
Oct 30 14:09:54.183471 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:54.183431    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75e20d5d-abc6-4682-8b47-0bb3abf6a790-openshift-service-ca" pod "75e20d5d-abc6-4682-8b47-0bb3abf6a790" (UID: "75e20d5d-abc6-4682-8b47-0bb3abf6a790"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:09:54.202691 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:54.185918    5093 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75e20d5d-abc6-4682-8b47-0bb3abf6a790-openshift-service-ca\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:09:54.202691 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:54.186056    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75e20d5d-abc6-4682-8b47-0bb3abf6a790-metrics-certs" pod "75e20d5d-abc6-4682-8b47-0bb3abf6a790" (UID: "75e20d5d-abc6-4682-8b47-0bb3abf6a790"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:09:54.202691 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:54.189604    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75e20d5d-abc6-4682-8b47-0bb3abf6a790-maas-api-tls" pod "75e20d5d-abc6-4682-8b47-0bb3abf6a790" (UID: "75e20d5d-abc6-4682-8b47-0bb3abf6a790"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:09:54.202691 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:54.190067    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75e20d5d-abc6-4682-8b47-0bb3abf6a790-kube-api-access-4j2mm" pod "75e20d5d-abc6-4682-8b47-0bb3abf6a790" (UID: "75e20d5d-abc6-4682-8b47-0bb3abf6a790"). InnerVolumeSpecName "kube-api-access-4j2mm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:09:54.287189 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:54.287105    5093 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75e20d5d-abc6-4682-8b47-0bb3abf6a790-metrics-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:09:54.287189 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:54.287189    5093 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75e20d5d-abc6-4682-8b47-0bb3abf6a790-maas-api-tls\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:09:54.287431 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:54.287205    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j2mm\" (UniqueName: \"kubernetes.io/projected/75e20d5d-abc6-4682-8b47-0bb3abf6a790-kube-api-access-4j2mm\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:09:54.718428 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:54.718395    5093 generic.go:356] "Generic (PLEG): container finished" podID="75e20d5d-abc6-4682-8b47-0bb3abf6a790" containerID="d92274fdca9a94d2ec274c8f78e377b9d7b83c0ba5b97a70f1251764469cd9dd" exitCode=0
Oct 30 14:09:54.718634 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:54.718439    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9bf8ad-6d48f89c8c-wzj8b"
Oct 30 14:09:54.718634 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:54.718477    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9bf8ad-6d48f89c8c-wzj8b" event={"ID":"75e20d5d-abc6-4682-8b47-0bb3abf6a790","Type":"ContainerDied","Data":"d92274fdca9a94d2ec274c8f78e377b9d7b83c0ba5b97a70f1251764469cd9dd"}
Oct 30 14:09:54.718634 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:54.718537    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9bf8ad-6d48f89c8c-wzj8b" event={"ID":"75e20d5d-abc6-4682-8b47-0bb3abf6a790","Type":"ContainerDied","Data":"8d2b91d83610aab524e1a862e08daa4032ccc6bb9dc362a5c4950c5af534e154"}
Oct 30 14:09:54.718634 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:54.718560    5093 scope.go:122] "RemoveContainer" containerID="d92274fdca9a94d2ec274c8f78e377b9d7b83c0ba5b97a70f1251764469cd9dd"
Oct 30 14:09:54.730339 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:54.730318    5093 scope.go:122] "RemoveContainer" containerID="d92274fdca9a94d2ec274c8f78e377b9d7b83c0ba5b97a70f1251764469cd9dd"
Oct 30 14:09:54.730675 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:09:54.730651    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d92274fdca9a94d2ec274c8f78e377b9d7b83c0ba5b97a70f1251764469cd9dd\": container with ID starting with d92274fdca9a94d2ec274c8f78e377b9d7b83c0ba5b97a70f1251764469cd9dd not found: ID does not exist" containerID="d92274fdca9a94d2ec274c8f78e377b9d7b83c0ba5b97a70f1251764469cd9dd"
Oct 30 14:09:54.730783 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:54.730682    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d92274fdca9a94d2ec274c8f78e377b9d7b83c0ba5b97a70f1251764469cd9dd"} err="failed to get container status \"d92274fdca9a94d2ec274c8f78e377b9d7b83c0ba5b97a70f1251764469cd9dd\": rpc error: code = NotFound desc = could not find container \"d92274fdca9a94d2ec274c8f78e377b9d7b83c0ba5b97a70f1251764469cd9dd\": container with ID starting with d92274fdca9a94d2ec274c8f78e377b9d7b83c0ba5b97a70f1251764469cd9dd not found: ID does not exist"
Oct 30 14:09:54.773199 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:54.773156    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-9bf8ad-6d48f89c8c-wzj8b"]
Oct 30 14:09:54.786055 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:54.786012    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-9bf8ad-6d48f89c8c-wzj8b"]
Oct 30 14:09:55.000666 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:09:55.000565    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75e20d5d-abc6-4682-8b47-0bb3abf6a790" path="/var/lib/kubelet/pods/75e20d5d-abc6-4682-8b47-0bb3abf6a790/volumes"
Oct 30 14:10:18.638785 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:18.638744    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-51dc9a/rate-test-model-51dc9a-kserve-69b7d876df-gqk6q"]
Oct 30 14:10:21.170966 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:10:21.170920    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 14:10:31.946752 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:31.946714    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7686-d1f729"]
Oct 30 14:10:31.947300 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:31.947285    5093 cpu_manager.go:414] "containerMap: removing container" podUID="3a707991-287b-4944-a992-72e750b13a89" containerName="payload-pre-processing"
Oct 30 14:10:31.947352 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:31.947301    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a707991-287b-4944-a992-72e750b13a89" containerName="payload-pre-processing"
Oct 30 14:10:31.947352 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:31.947318    5093 cpu_manager.go:414] "containerMap: removing container" podUID="a50e3626-0c42-47fc-83c3-a96c1889335c" containerName="maas-api"
Oct 30 14:10:31.947352 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:31.947325    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a50e3626-0c42-47fc-83c3-a96c1889335c" containerName="maas-api"
Oct 30 14:10:31.947352 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:31.947332    5093 cpu_manager.go:414] "containerMap: removing container" podUID="4b8aed42-a18f-4cb3-aa5d-7a3cf0345391" containerName="payload-processing"
Oct 30 14:10:31.947352 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:31.947337    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b8aed42-a18f-4cb3-aa5d-7a3cf0345391" containerName="payload-processing"
Oct 30 14:10:31.947513 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:31.947355    5093 cpu_manager.go:414] "containerMap: removing container" podUID="d6a44981-104f-4d3e-9937-d184a6ddc482" containerName="main"
Oct 30 14:10:31.947513 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:31.947360    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6a44981-104f-4d3e-9937-d184a6ddc482" containerName="main"
Oct 30 14:10:31.947513 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:31.947375    5093 cpu_manager.go:414] "containerMap: removing container" podUID="75e20d5d-abc6-4682-8b47-0bb3abf6a790" containerName="maas-api"
Oct 30 14:10:31.947513 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:31.947380    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75e20d5d-abc6-4682-8b47-0bb3abf6a790" containerName="maas-api"
Oct 30 14:10:31.947513 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:31.947389    5093 cpu_manager.go:414] "containerMap: removing container" podUID="0d63ba90-2e39-44a9-aa12-ec1adf2dc23f" containerName="maas-api"
Oct 30 14:10:31.947513 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:31.947394    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d63ba90-2e39-44a9-aa12-ec1adf2dc23f" containerName="maas-api"
Oct 30 14:10:31.947513 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:31.947405    5093 cpu_manager.go:414] "containerMap: removing container" podUID="ba156eeb-9955-4f9a-857b-2c4d502a5fcc" containerName="main"
Oct 30 14:10:31.947513 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:31.947409    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba156eeb-9955-4f9a-857b-2c4d502a5fcc" containerName="main"
Oct 30 14:10:31.947513 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:31.947423    5093 cpu_manager.go:414] "containerMap: removing container" podUID="c4b4e00e-f58d-4a4e-b7db-19f724360db1" containerName="main"
Oct 30 14:10:31.947513 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:31.947428    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4b4e00e-f58d-4a4e-b7db-19f724360db1" containerName="main"
Oct 30 14:10:31.947513 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:31.947507    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="a50e3626-0c42-47fc-83c3-a96c1889335c" containerName="maas-api"
Oct 30 14:10:31.947895 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:31.947517    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b8aed42-a18f-4cb3-aa5d-7a3cf0345391" containerName="payload-processing"
Oct 30 14:10:31.947895 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:31.947526    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6a44981-104f-4d3e-9937-d184a6ddc482" containerName="main"
Oct 30 14:10:31.947895 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:31.947535    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="75e20d5d-abc6-4682-8b47-0bb3abf6a790" containerName="maas-api"
Oct 30 14:10:31.947895 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:31.947542    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d63ba90-2e39-44a9-aa12-ec1adf2dc23f" containerName="maas-api"
Oct 30 14:10:31.947895 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:31.947549    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba156eeb-9955-4f9a-857b-2c4d502a5fcc" containerName="main"
Oct 30 14:10:31.947895 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:31.947556    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4b4e00e-f58d-4a4e-b7db-19f724360db1" containerName="main"
Oct 30 14:10:31.947895 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:31.947562    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a707991-287b-4944-a992-72e750b13a89" containerName="payload-pre-processing"
Oct 30 14:10:31.951224 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:31.951202    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-d1f729"
Oct 30 14:10:31.967085 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:31.967050    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7686-d1f729"]
Oct 30 14:10:31.974867 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:31.974830    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4bkkd\""
Oct 30 14:10:32.000770 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:32.000727    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rnlt\" (UniqueName: \"kubernetes.io/projected/0f5d82fb-8aab-473d-af1f-2154c4f5a49a-kube-api-access-6rnlt\") pod \"test-curl-7686-d1f729\" (UID: \"0f5d82fb-8aab-473d-af1f-2154c4f5a49a\") " pod="opendatahub/test-curl-7686-d1f729"
Oct 30 14:10:32.102432 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:32.102382    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rnlt\" (UniqueName: \"kubernetes.io/projected/0f5d82fb-8aab-473d-af1f-2154c4f5a49a-kube-api-access-6rnlt\") pod \"test-curl-7686-d1f729\" (UID: \"0f5d82fb-8aab-473d-af1f-2154c4f5a49a\") " pod="opendatahub/test-curl-7686-d1f729"
Oct 30 14:10:32.122698 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:32.122663    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rnlt\" (UniqueName: \"kubernetes.io/projected/0f5d82fb-8aab-473d-af1f-2154c4f5a49a-kube-api-access-6rnlt\") pod \"test-curl-7686-d1f729\" (UID: \"0f5d82fb-8aab-473d-af1f-2154c4f5a49a\") " pod="opendatahub/test-curl-7686-d1f729"
Oct 30 14:10:32.266001 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:32.265891    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-d1f729"
Oct 30 14:10:32.565844 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:32.565744    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7686-d1f729"]
Oct 30 14:10:32.577227 ip-10-0-134-31 kubenswrapper[5093]: W1030 14:10:32.577190    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f5d82fb_8aab_473d_af1f_2154c4f5a49a.slice/crio-f5e2bc765b53817bc484e5153623dd054a8b4c1ddb674cebf50523a7058e02cf WatchSource:0}: Error finding container f5e2bc765b53817bc484e5153623dd054a8b4c1ddb674cebf50523a7058e02cf: Status 404 returned error can't find the container with id f5e2bc765b53817bc484e5153623dd054a8b4c1ddb674cebf50523a7058e02cf
Oct 30 14:10:33.011777 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:33.011733    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7686-d1f729" event={"ID":"0f5d82fb-8aab-473d-af1f-2154c4f5a49a","Type":"ContainerStarted","Data":"94ddf4454aa6551d368c0193700fe10382c588181f71b6c3c1e245de3a59ec81"}
Oct 30 14:10:33.012285 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:33.011787    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7686-d1f729" event={"ID":"0f5d82fb-8aab-473d-af1f-2154c4f5a49a","Type":"ContainerStarted","Data":"f5e2bc765b53817bc484e5153623dd054a8b4c1ddb674cebf50523a7058e02cf"}
Oct 30 14:10:33.054070 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:33.053995    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7686-d1f729" podStartSLOduration=2.053979847 podStartE2EDuration="2.053979847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:10:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:10:33.049764227 +0000 UTC m=+4052.197031468" watchObservedRunningTime="2026-10-30 14:10:33.053979847 +0000 UTC m=+4052.201247088"
Oct 30 14:10:33.912928 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:33.912859    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-51dc9a/rate-test-model-51dc9a-kserve-69b7d876df-gqk6q" podUID="16483221-0a88-4aa3-a47d-6d119206b423" containerName="main" containerID="cri-o://35252b57dac76b6bcfda78df08f1f4ea76f7db39fb47c4b3f72a88771d491504" gracePeriod=45
Oct 30 14:10:33.941113 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:33.941068    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7686-d1f729"]
Oct 30 14:10:33.945311 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:33.945273    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7686-d1f729"]
Oct 30 14:10:34.117290 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:34.117249    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7686-4ba6d9"]
Oct 30 14:10:34.117878 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:34.117863    5093 cpu_manager.go:414] "containerMap: removing container" podUID="0f5d82fb-8aab-473d-af1f-2154c4f5a49a" containerName="test-curl-7686-d1f729"
Oct 30 14:10:34.117929 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:34.117879    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f5d82fb-8aab-473d-af1f-2154c4f5a49a" containerName="test-curl-7686-d1f729"
Oct 30 14:10:34.118003 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:34.117992    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f5d82fb-8aab-473d-af1f-2154c4f5a49a" containerName="test-curl-7686-d1f729"
Oct 30 14:10:34.122276 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:34.122245    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-4ba6d9"
Oct 30 14:10:34.142146 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:34.142069    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7686-4ba6d9"]
Oct 30 14:10:34.238675 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:34.238647    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-51dc9a/rate-test-model-51dc9a-kserve-69b7d876df-gqk6q"
Oct 30 14:10:34.241324 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:34.241292    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz98h\" (UniqueName: \"kubernetes.io/projected/a51f3807-9699-4d02-8839-302a7f44afa3-kube-api-access-hz98h\") pod \"test-curl-7686-4ba6d9\" (UID: \"a51f3807-9699-4d02-8839-302a7f44afa3\") " pod="opendatahub/test-curl-7686-4ba6d9"
Oct 30 14:10:34.342262 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:34.342222    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16483221-0a88-4aa3-a47d-6d119206b423-kube-api-access-w5ckl\" (UniqueName: \"kubernetes.io/projected/16483221-0a88-4aa3-a47d-6d119206b423-kube-api-access-w5ckl\") pod \"16483221-0a88-4aa3-a47d-6d119206b423\" (UID: \"16483221-0a88-4aa3-a47d-6d119206b423\") "
Oct 30 14:10:34.342485 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:34.342288    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16483221-0a88-4aa3-a47d-6d119206b423-home\" (UniqueName: \"kubernetes.io/empty-dir/16483221-0a88-4aa3-a47d-6d119206b423-home\") pod \"16483221-0a88-4aa3-a47d-6d119206b423\" (UID: \"16483221-0a88-4aa3-a47d-6d119206b423\") "
Oct 30 14:10:34.342485 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:34.342333    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16483221-0a88-4aa3-a47d-6d119206b423-dshm\" (UniqueName: \"kubernetes.io/empty-dir/16483221-0a88-4aa3-a47d-6d119206b423-dshm\") pod \"16483221-0a88-4aa3-a47d-6d119206b423\" (UID: \"16483221-0a88-4aa3-a47d-6d119206b423\") "
Oct 30 14:10:34.342623 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:34.342511    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16483221-0a88-4aa3-a47d-6d119206b423-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16483221-0a88-4aa3-a47d-6d119206b423-model-cache\") pod \"16483221-0a88-4aa3-a47d-6d119206b423\" (UID: \"16483221-0a88-4aa3-a47d-6d119206b423\") "
Oct 30 14:10:34.342697 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:34.342650    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16483221-0a88-4aa3-a47d-6d119206b423-home" pod "16483221-0a88-4aa3-a47d-6d119206b423" (UID: "16483221-0a88-4aa3-a47d-6d119206b423"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:10:34.342825 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:34.342797    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16483221-0a88-4aa3-a47d-6d119206b423-model-cache" pod "16483221-0a88-4aa3-a47d-6d119206b423" (UID: "16483221-0a88-4aa3-a47d-6d119206b423"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:10:34.342942 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:34.342924    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16483221-0a88-4aa3-a47d-6d119206b423-tls-certs\" (UniqueName: \"kubernetes.io/secret/16483221-0a88-4aa3-a47d-6d119206b423-tls-certs\") pod \"16483221-0a88-4aa3-a47d-6d119206b423\" (UID: \"16483221-0a88-4aa3-a47d-6d119206b423\") "
Oct 30 14:10:34.343039 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:34.343022    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16483221-0a88-4aa3-a47d-6d119206b423-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16483221-0a88-4aa3-a47d-6d119206b423-tmp-dir\") pod \"16483221-0a88-4aa3-a47d-6d119206b423\" (UID: \"16483221-0a88-4aa3-a47d-6d119206b423\") "
Oct 30 14:10:34.343379 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:34.343348    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16483221-0a88-4aa3-a47d-6d119206b423-tmp-dir" pod "16483221-0a88-4aa3-a47d-6d119206b423" (UID: "16483221-0a88-4aa3-a47d-6d119206b423"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:10:34.343525 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:34.343499    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz98h\" (UniqueName: \"kubernetes.io/projected/a51f3807-9699-4d02-8839-302a7f44afa3-kube-api-access-hz98h\") pod \"test-curl-7686-4ba6d9\" (UID: \"a51f3807-9699-4d02-8839-302a7f44afa3\") " pod="opendatahub/test-curl-7686-4ba6d9"
Oct 30 14:10:34.343718 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:34.343702    5093 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16483221-0a88-4aa3-a47d-6d119206b423-model-cache\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:34.343809 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:34.343721    5093 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16483221-0a88-4aa3-a47d-6d119206b423-tmp-dir\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:34.343809 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:34.343736    5093 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16483221-0a88-4aa3-a47d-6d119206b423-home\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:34.345020 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:34.344990    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16483221-0a88-4aa3-a47d-6d119206b423-dshm" pod "16483221-0a88-4aa3-a47d-6d119206b423" (UID: "16483221-0a88-4aa3-a47d-6d119206b423"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:10:34.345188 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:34.345124    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16483221-0a88-4aa3-a47d-6d119206b423-kube-api-access-w5ckl" pod "16483221-0a88-4aa3-a47d-6d119206b423" (UID: "16483221-0a88-4aa3-a47d-6d119206b423"). InnerVolumeSpecName "kube-api-access-w5ckl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:10:34.345587 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:34.345560    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16483221-0a88-4aa3-a47d-6d119206b423-tls-certs" pod "16483221-0a88-4aa3-a47d-6d119206b423" (UID: "16483221-0a88-4aa3-a47d-6d119206b423"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:10:34.364160 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:34.364109    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz98h\" (UniqueName: \"kubernetes.io/projected/a51f3807-9699-4d02-8839-302a7f44afa3-kube-api-access-hz98h\") pod \"test-curl-7686-4ba6d9\" (UID: \"a51f3807-9699-4d02-8839-302a7f44afa3\") " pod="opendatahub/test-curl-7686-4ba6d9"
Oct 30 14:10:34.445223 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:34.445177    5093 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16483221-0a88-4aa3-a47d-6d119206b423-tls-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:34.445223 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:34.445218    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5ckl\" (UniqueName: \"kubernetes.io/projected/16483221-0a88-4aa3-a47d-6d119206b423-kube-api-access-w5ckl\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:34.445223 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:34.445230    5093 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16483221-0a88-4aa3-a47d-6d119206b423-dshm\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:34.452818 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:34.452785    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-4ba6d9"
Oct 30 14:10:34.749876 ip-10-0-134-31 kubenswrapper[5093]: W1030 14:10:34.749842    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda51f3807_9699_4d02_8839_302a7f44afa3.slice/crio-02eecf9c5aceaacec857cc0d8652a6a97fbe1f9e123d06e5ae2de178fc2af413 WatchSource:0}: Error finding container 02eecf9c5aceaacec857cc0d8652a6a97fbe1f9e123d06e5ae2de178fc2af413: Status 404 returned error can't find the container with id 02eecf9c5aceaacec857cc0d8652a6a97fbe1f9e123d06e5ae2de178fc2af413
Oct 30 14:10:34.750266 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:34.750242    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7686-4ba6d9"]
Oct 30 14:10:35.027144 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:35.027091    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b"]
Oct 30 14:10:35.027375 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:35.027346    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b" podUID="762bb1a6-9339-4c49-bf97-1e160e7b0fc6" containerName="istio-proxy" containerID="cri-o://b3e545a30c051f6f4a59c8ea60a716fc6d246eac9915f0c924fcb1ac06cb46bb" gracePeriod=30
Oct 30 14:10:35.028039 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:35.027834    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7686-4ba6d9" event={"ID":"a51f3807-9699-4d02-8839-302a7f44afa3","Type":"ContainerStarted","Data":"a494db7165dfb491dd0646e2431e167e2c723f497ed906e844d33da63a91dd4d"}
Oct 30 14:10:35.028039 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:35.027875    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7686-4ba6d9" event={"ID":"a51f3807-9699-4d02-8839-302a7f44afa3","Type":"ContainerStarted","Data":"02eecf9c5aceaacec857cc0d8652a6a97fbe1f9e123d06e5ae2de178fc2af413"}
Oct 30 14:10:35.029502 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:35.029480    5093 generic.go:356] "Generic (PLEG): container finished" podID="16483221-0a88-4aa3-a47d-6d119206b423" containerID="35252b57dac76b6bcfda78df08f1f4ea76f7db39fb47c4b3f72a88771d491504" exitCode=0
Oct 30 14:10:35.029612 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:35.029520    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-51dc9a/rate-test-model-51dc9a-kserve-69b7d876df-gqk6q"
Oct 30 14:10:35.029612 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:35.029554    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-51dc9a/rate-test-model-51dc9a-kserve-69b7d876df-gqk6q" event={"ID":"16483221-0a88-4aa3-a47d-6d119206b423","Type":"ContainerDied","Data":"35252b57dac76b6bcfda78df08f1f4ea76f7db39fb47c4b3f72a88771d491504"}
Oct 30 14:10:35.029612 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:35.029593    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-51dc9a/rate-test-model-51dc9a-kserve-69b7d876df-gqk6q" event={"ID":"16483221-0a88-4aa3-a47d-6d119206b423","Type":"ContainerDied","Data":"65f155680c4da839b8193e02eb304be5cfd20a091494a35ab690f79c5232f1b7"}
Oct 30 14:10:35.029769 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:35.029619    5093 scope.go:122] "RemoveContainer" containerID="35252b57dac76b6bcfda78df08f1f4ea76f7db39fb47c4b3f72a88771d491504"
Oct 30 14:10:35.029834 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:35.029780    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7686-d1f729" podUID="0f5d82fb-8aab-473d-af1f-2154c4f5a49a" containerName="test-curl-7686-d1f729" containerID="cri-o://94ddf4454aa6551d368c0193700fe10382c588181f71b6c3c1e245de3a59ec81" gracePeriod=2
Oct 30 14:10:35.045085 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:35.041180    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2"]
Oct 30 14:10:35.045085 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:35.041432    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2" podUID="d2c83cc2-4c02-493a-948a-3a0ecf046a37" containerName="istio-proxy" containerID="cri-o://8234e542f8c3186d92daa8ef0662dbea057aa22f4553b94d48752bdc7dddd7cb" gracePeriod=30
Oct 30 14:10:35.045085 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:35.042799    5093 scope.go:122] "RemoveContainer" containerID="35252b57dac76b6bcfda78df08f1f4ea76f7db39fb47c4b3f72a88771d491504"
Oct 30 14:10:35.045085 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:10:35.044009    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35252b57dac76b6bcfda78df08f1f4ea76f7db39fb47c4b3f72a88771d491504\": container with ID starting with 35252b57dac76b6bcfda78df08f1f4ea76f7db39fb47c4b3f72a88771d491504 not found: ID does not exist" containerID="35252b57dac76b6bcfda78df08f1f4ea76f7db39fb47c4b3f72a88771d491504"
Oct 30 14:10:35.045085 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:35.044049    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35252b57dac76b6bcfda78df08f1f4ea76f7db39fb47c4b3f72a88771d491504"} err="failed to get container status \"35252b57dac76b6bcfda78df08f1f4ea76f7db39fb47c4b3f72a88771d491504\": rpc error: code = NotFound desc = could not find container \"35252b57dac76b6bcfda78df08f1f4ea76f7db39fb47c4b3f72a88771d491504\": container with ID starting with 35252b57dac76b6bcfda78df08f1f4ea76f7db39fb47c4b3f72a88771d491504 not found: ID does not exist"
Oct 30 14:10:35.046042 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:35.046008    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2"]
Oct 30 14:10:35.046323 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:35.046291    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2" podUID="926c412f-623c-489d-8100-c66c75a5f478" containerName="istio-proxy" containerID="cri-o://1b59835711224dbaf4410d797ab735072aa6903fe877d6734184f207c8ed1bc8" gracePeriod=30
Oct 30 14:10:35.049383 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:35.049329    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5"]
Oct 30 14:10:35.049625 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:35.049598    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5" podUID="f9778494-18c5-4eb6-926b-92f69cbde312" containerName="istio-proxy" containerID="cri-o://36d337dfb735313c530f1c21e21c4e270679971d466c5794a3eb797d63bb2360" gracePeriod=30
Oct 30 14:10:35.149926 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:35.149873    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-51dc9a/rate-test-model-51dc9a-kserve-69b7d876df-gqk6q" podUID="16483221-0a88-4aa3-a47d-6d119206b423" containerName="main" probeResult="failure" output="Get \"https://10.132.0.202:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:10:35.172765 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:35.172697    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7686-4ba6d9" podStartSLOduration=1.1726793500000001 podStartE2EDuration="1.17267935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:10:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:10:35.168810334 +0000 UTC m=+4054.316077574" watchObservedRunningTime="2026-10-30 14:10:35.17267935 +0000 UTC m=+4054.319946593"
Oct 30 14:10:35.292593 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:35.292497    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-51dc9a/rate-test-model-51dc9a-kserve-69b7d876df-gqk6q"]
Oct 30 14:10:35.297038 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:35.297007    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-51dc9a/rate-test-model-51dc9a-kserve-69b7d876df-gqk6q"]
Oct 30 14:10:36.067931 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:36.067897    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7686-4ba6d9"]
Oct 30 14:10:36.071358 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:36.071325    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7686-4ba6d9"]
Oct 30 14:10:36.999513 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:36.999475    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16483221-0a88-4aa3-a47d-6d119206b423" path="/var/lib/kubelet/pods/16483221-0a88-4aa3-a47d-6d119206b423/volumes"
Oct 30 14:10:37.044878 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:37.044840    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7686-4ba6d9" podUID="a51f3807-9699-4d02-8839-302a7f44afa3" containerName="test-curl-7686-4ba6d9" containerID="cri-o://a494db7165dfb491dd0646e2431e167e2c723f497ed906e844d33da63a91dd4d" gracePeriod=2
Oct 30 14:10:37.264443 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:37.264377    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-d1f729"
Oct 30 14:10:37.382732 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:37.382688    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f5d82fb-8aab-473d-af1f-2154c4f5a49a-kube-api-access-6rnlt\" (UniqueName: \"kubernetes.io/projected/0f5d82fb-8aab-473d-af1f-2154c4f5a49a-kube-api-access-6rnlt\") pod \"0f5d82fb-8aab-473d-af1f-2154c4f5a49a\" (UID: \"0f5d82fb-8aab-473d-af1f-2154c4f5a49a\") "
Oct 30 14:10:37.384976 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:37.384942    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f5d82fb-8aab-473d-af1f-2154c4f5a49a-kube-api-access-6rnlt" pod "0f5d82fb-8aab-473d-af1f-2154c4f5a49a" (UID: "0f5d82fb-8aab-473d-af1f-2154c4f5a49a"). InnerVolumeSpecName "kube-api-access-6rnlt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:10:37.484589 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:37.484551    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rnlt\" (UniqueName: \"kubernetes.io/projected/0f5d82fb-8aab-473d-af1f-2154c4f5a49a-kube-api-access-6rnlt\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:38.054284 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:38.054242    5093 generic.go:356] "Generic (PLEG): container finished" podID="0f5d82fb-8aab-473d-af1f-2154c4f5a49a" containerID="94ddf4454aa6551d368c0193700fe10382c588181f71b6c3c1e245de3a59ec81" exitCode=137
Oct 30 14:10:38.054809 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:38.054294    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-d1f729"
Oct 30 14:10:38.054932 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:38.054908    5093 scope.go:122] "RemoveContainer" containerID="94ddf4454aa6551d368c0193700fe10382c588181f71b6c3c1e245de3a59ec81"
Oct 30 14:10:38.068302 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:38.068274    5093 scope.go:122] "RemoveContainer" containerID="94ddf4454aa6551d368c0193700fe10382c588181f71b6c3c1e245de3a59ec81"
Oct 30 14:10:38.068653 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:10:38.068629    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94ddf4454aa6551d368c0193700fe10382c588181f71b6c3c1e245de3a59ec81\": container with ID starting with 94ddf4454aa6551d368c0193700fe10382c588181f71b6c3c1e245de3a59ec81 not found: ID does not exist" containerID="94ddf4454aa6551d368c0193700fe10382c588181f71b6c3c1e245de3a59ec81"
Oct 30 14:10:38.068736 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:38.068666    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94ddf4454aa6551d368c0193700fe10382c588181f71b6c3c1e245de3a59ec81"} err="failed to get container status \"94ddf4454aa6551d368c0193700fe10382c588181f71b6c3c1e245de3a59ec81\": rpc error: code = NotFound desc = could not find container \"94ddf4454aa6551d368c0193700fe10382c588181f71b6c3c1e245de3a59ec81\": container with ID starting with 94ddf4454aa6551d368c0193700fe10382c588181f71b6c3c1e245de3a59ec81 not found: ID does not exist"
Oct 30 14:10:38.079782 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:10:38.079739    5093 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7686-d1f729\" is forbidden: User \"system:node:ip-10-0-134-31.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-31.ec2.internal' and this object" podUID="0f5d82fb-8aab-473d-af1f-2154c4f5a49a" pod="opendatahub/test-curl-7686-d1f729"
Oct 30 14:10:38.999584 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:38.999547    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f5d82fb-8aab-473d-af1f-2154c4f5a49a" path="/var/lib/kubelet/pods/0f5d82fb-8aab-473d-af1f-2154c4f5a49a/volumes"
Oct 30 14:10:39.264150 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:39.264066    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-4ba6d9"
Oct 30 14:10:39.407214 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:39.407164    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a51f3807-9699-4d02-8839-302a7f44afa3-kube-api-access-hz98h\" (UniqueName: \"kubernetes.io/projected/a51f3807-9699-4d02-8839-302a7f44afa3-kube-api-access-hz98h\") pod \"a51f3807-9699-4d02-8839-302a7f44afa3\" (UID: \"a51f3807-9699-4d02-8839-302a7f44afa3\") "
Oct 30 14:10:39.409454 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:39.409424    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a51f3807-9699-4d02-8839-302a7f44afa3-kube-api-access-hz98h" pod "a51f3807-9699-4d02-8839-302a7f44afa3" (UID: "a51f3807-9699-4d02-8839-302a7f44afa3"). InnerVolumeSpecName "kube-api-access-hz98h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:10:39.508675 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:39.508638    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz98h\" (UniqueName: \"kubernetes.io/projected/a51f3807-9699-4d02-8839-302a7f44afa3-kube-api-access-hz98h\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:40.076225 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.076183    5093 generic.go:356] "Generic (PLEG): container finished" podID="a51f3807-9699-4d02-8839-302a7f44afa3" containerID="a494db7165dfb491dd0646e2431e167e2c723f497ed906e844d33da63a91dd4d" exitCode=137
Oct 30 14:10:40.076389 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.076237    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-4ba6d9"
Oct 30 14:10:40.076389 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.076277    5093 scope.go:122] "RemoveContainer" containerID="a494db7165dfb491dd0646e2431e167e2c723f497ed906e844d33da63a91dd4d"
Oct 30 14:10:40.296340 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.296090    5093 scope.go:122] "RemoveContainer" containerID="a494db7165dfb491dd0646e2431e167e2c723f497ed906e844d33da63a91dd4d"
Oct 30 14:10:40.296751 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:10:40.296557    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a494db7165dfb491dd0646e2431e167e2c723f497ed906e844d33da63a91dd4d\": container with ID starting with a494db7165dfb491dd0646e2431e167e2c723f497ed906e844d33da63a91dd4d not found: ID does not exist" containerID="a494db7165dfb491dd0646e2431e167e2c723f497ed906e844d33da63a91dd4d"
Oct 30 14:10:40.296751 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.296601    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a494db7165dfb491dd0646e2431e167e2c723f497ed906e844d33da63a91dd4d"} err="failed to get container status \"a494db7165dfb491dd0646e2431e167e2c723f497ed906e844d33da63a91dd4d\": rpc error: code = NotFound desc = could not find container \"a494db7165dfb491dd0646e2431e167e2c723f497ed906e844d33da63a91dd4d\": container with ID starting with a494db7165dfb491dd0646e2431e167e2c723f497ed906e844d33da63a91dd4d not found: ID does not exist"
Oct 30 14:10:40.301496 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:10:40.301415    5093 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7686-4ba6d9\" is forbidden: User \"system:node:ip-10-0-134-31.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-31.ec2.internal' and this object" podUID="a51f3807-9699-4d02-8839-302a7f44afa3" pod="opendatahub/test-curl-7686-4ba6d9"
Oct 30 14:10:40.394417 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.394383    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2"
Oct 30 14:10:40.408747 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:10:40.408706    5093 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7686-4ba6d9\" is forbidden: User \"system:node:ip-10-0-134-31.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-31.ec2.internal' and this object" podUID="a51f3807-9699-4d02-8839-302a7f44afa3" pod="opendatahub/test-curl-7686-4ba6d9"
Oct 30 14:10:40.448179 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.443434    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2"
Oct 30 14:10:40.496538 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:10:40.496497    5093 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7686-4ba6d9\" is forbidden: User \"system:node:ip-10-0-134-31.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-31.ec2.internal' and this object" podUID="a51f3807-9699-4d02-8839-302a7f44afa3" pod="opendatahub/test-curl-7686-4ba6d9"
Oct 30 14:10:40.521145 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.521103    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/926c412f-623c-489d-8100-c66c75a5f478-kube-api-access-djmrs\" (UniqueName: \"kubernetes.io/projected/926c412f-623c-489d-8100-c66c75a5f478-kube-api-access-djmrs\") pod \"926c412f-623c-489d-8100-c66c75a5f478\" (UID: \"926c412f-623c-489d-8100-c66c75a5f478\") "
Oct 30 14:10:40.521340 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.521189    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/926c412f-623c-489d-8100-c66c75a5f478-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/926c412f-623c-489d-8100-c66c75a5f478-istio-envoy\") pod \"926c412f-623c-489d-8100-c66c75a5f478\" (UID: \"926c412f-623c-489d-8100-c66c75a5f478\") "
Oct 30 14:10:40.521340 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.521238    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/926c412f-623c-489d-8100-c66c75a5f478-istio-token\" (UniqueName: \"kubernetes.io/projected/926c412f-623c-489d-8100-c66c75a5f478-istio-token\") pod \"926c412f-623c-489d-8100-c66c75a5f478\" (UID: \"926c412f-623c-489d-8100-c66c75a5f478\") "
Oct 30 14:10:40.521340 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.521293    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/926c412f-623c-489d-8100-c66c75a5f478-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/926c412f-623c-489d-8100-c66c75a5f478-workload-certs\") pod \"926c412f-623c-489d-8100-c66c75a5f478\" (UID: \"926c412f-623c-489d-8100-c66c75a5f478\") "
Oct 30 14:10:40.521631 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.521387    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/926c412f-623c-489d-8100-c66c75a5f478-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/926c412f-623c-489d-8100-c66c75a5f478-credential-socket\") pod \"926c412f-623c-489d-8100-c66c75a5f478\" (UID: \"926c412f-623c-489d-8100-c66c75a5f478\") "
Oct 30 14:10:40.521631 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.521473    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/926c412f-623c-489d-8100-c66c75a5f478-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/926c412f-623c-489d-8100-c66c75a5f478-istiod-ca-cert\") pod \"926c412f-623c-489d-8100-c66c75a5f478\" (UID: \"926c412f-623c-489d-8100-c66c75a5f478\") "
Oct 30 14:10:40.521631 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.521509    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/926c412f-623c-489d-8100-c66c75a5f478-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/926c412f-623c-489d-8100-c66c75a5f478-istio-podinfo\") pod \"926c412f-623c-489d-8100-c66c75a5f478\" (UID: \"926c412f-623c-489d-8100-c66c75a5f478\") "
Oct 30 14:10:40.521631 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.521617    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/926c412f-623c-489d-8100-c66c75a5f478-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/926c412f-623c-489d-8100-c66c75a5f478-workload-socket\") pod \"926c412f-623c-489d-8100-c66c75a5f478\" (UID: \"926c412f-623c-489d-8100-c66c75a5f478\") "
Oct 30 14:10:40.521961 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.521658    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/926c412f-623c-489d-8100-c66c75a5f478-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/926c412f-623c-489d-8100-c66c75a5f478-istio-data\") pod \"926c412f-623c-489d-8100-c66c75a5f478\" (UID: \"926c412f-623c-489d-8100-c66c75a5f478\") "
Oct 30 14:10:40.522331 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.522298    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/926c412f-623c-489d-8100-c66c75a5f478-workload-certs" pod "926c412f-623c-489d-8100-c66c75a5f478" (UID: "926c412f-623c-489d-8100-c66c75a5f478"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:10:40.522462 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.522442    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/926c412f-623c-489d-8100-c66c75a5f478-istio-data" pod "926c412f-623c-489d-8100-c66c75a5f478" (UID: "926c412f-623c-489d-8100-c66c75a5f478"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:10:40.522635 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.522610    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/926c412f-623c-489d-8100-c66c75a5f478-workload-socket" pod "926c412f-623c-489d-8100-c66c75a5f478" (UID: "926c412f-623c-489d-8100-c66c75a5f478"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:10:40.522968 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.522941    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/926c412f-623c-489d-8100-c66c75a5f478-credential-socket" pod "926c412f-623c-489d-8100-c66c75a5f478" (UID: "926c412f-623c-489d-8100-c66c75a5f478"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:10:40.523420 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.523385    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/926c412f-623c-489d-8100-c66c75a5f478-istiod-ca-cert" pod "926c412f-623c-489d-8100-c66c75a5f478" (UID: "926c412f-623c-489d-8100-c66c75a5f478"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:10:40.528196 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.527113    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/926c412f-623c-489d-8100-c66c75a5f478-istio-podinfo" pod "926c412f-623c-489d-8100-c66c75a5f478" (UID: "926c412f-623c-489d-8100-c66c75a5f478"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:10:40.529899 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.529870    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/926c412f-623c-489d-8100-c66c75a5f478-istio-envoy" pod "926c412f-623c-489d-8100-c66c75a5f478" (UID: "926c412f-623c-489d-8100-c66c75a5f478"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:10:40.530073 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.530052    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/926c412f-623c-489d-8100-c66c75a5f478-istio-token" pod "926c412f-623c-489d-8100-c66c75a5f478" (UID: "926c412f-623c-489d-8100-c66c75a5f478"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:10:40.531035 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.531008    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/926c412f-623c-489d-8100-c66c75a5f478-kube-api-access-djmrs" pod "926c412f-623c-489d-8100-c66c75a5f478" (UID: "926c412f-623c-489d-8100-c66c75a5f478"). InnerVolumeSpecName "kube-api-access-djmrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:10:40.532652 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.532629    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5"
Oct 30 14:10:40.535530 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.535512    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b"
Oct 30 14:10:40.556859 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:10:40.556743    5093 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7686-4ba6d9\" is forbidden: User \"system:node:ip-10-0-134-31.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-31.ec2.internal' and this object" podUID="a51f3807-9699-4d02-8839-302a7f44afa3" pod="opendatahub/test-curl-7686-4ba6d9"
Oct 30 14:10:40.623249 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.623217    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9778494-18c5-4eb6-926b-92f69cbde312-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9778494-18c5-4eb6-926b-92f69cbde312-credential-socket\") pod \"f9778494-18c5-4eb6-926b-92f69cbde312\" (UID: \"f9778494-18c5-4eb6-926b-92f69cbde312\") "
Oct 30 14:10:40.623249 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.623254    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2c83cc2-4c02-493a-948a-3a0ecf046a37-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c83cc2-4c02-493a-948a-3a0ecf046a37-workload-socket\") pod \"d2c83cc2-4c02-493a-948a-3a0ecf046a37\" (UID: \"d2c83cc2-4c02-493a-948a-3a0ecf046a37\") "
Oct 30 14:10:40.623514 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.623293    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2c83cc2-4c02-493a-948a-3a0ecf046a37-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c83cc2-4c02-493a-948a-3a0ecf046a37-credential-socket\") pod \"d2c83cc2-4c02-493a-948a-3a0ecf046a37\" (UID: \"d2c83cc2-4c02-493a-948a-3a0ecf046a37\") "
Oct 30 14:10:40.623514 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.623326    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9778494-18c5-4eb6-926b-92f69cbde312-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9778494-18c5-4eb6-926b-92f69cbde312-istio-envoy\") pod \"f9778494-18c5-4eb6-926b-92f69cbde312\" (UID: \"f9778494-18c5-4eb6-926b-92f69cbde312\") "
Oct 30 14:10:40.623514 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.623349    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9778494-18c5-4eb6-926b-92f69cbde312-kube-api-access-cnpl7\" (UniqueName: \"kubernetes.io/projected/f9778494-18c5-4eb6-926b-92f69cbde312-kube-api-access-cnpl7\") pod \"f9778494-18c5-4eb6-926b-92f69cbde312\" (UID: \"f9778494-18c5-4eb6-926b-92f69cbde312\") "
Oct 30 14:10:40.623690 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.623580    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c83cc2-4c02-493a-948a-3a0ecf046a37-workload-socket" pod "d2c83cc2-4c02-493a-948a-3a0ecf046a37" (UID: "d2c83cc2-4c02-493a-948a-3a0ecf046a37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:10:40.623690 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.623670    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c83cc2-4c02-493a-948a-3a0ecf046a37-credential-socket" pod "d2c83cc2-4c02-493a-948a-3a0ecf046a37" (UID: "d2c83cc2-4c02-493a-948a-3a0ecf046a37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:10:40.623837 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.623375    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9778494-18c5-4eb6-926b-92f69cbde312-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9778494-18c5-4eb6-926b-92f69cbde312-istio-data\") pod \"f9778494-18c5-4eb6-926b-92f69cbde312\" (UID: \"f9778494-18c5-4eb6-926b-92f69cbde312\") "
Oct 30 14:10:40.623972 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.623872    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2c83cc2-4c02-493a-948a-3a0ecf046a37-istio-token\" (UniqueName: \"kubernetes.io/projected/d2c83cc2-4c02-493a-948a-3a0ecf046a37-istio-token\") pod \"d2c83cc2-4c02-493a-948a-3a0ecf046a37\" (UID: \"d2c83cc2-4c02-493a-948a-3a0ecf046a37\") "
Oct 30 14:10:40.623972 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.623583    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9778494-18c5-4eb6-926b-92f69cbde312-credential-socket" pod "f9778494-18c5-4eb6-926b-92f69cbde312" (UID: "f9778494-18c5-4eb6-926b-92f69cbde312"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:10:40.623972 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.623824    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9778494-18c5-4eb6-926b-92f69cbde312-istio-data" pod "f9778494-18c5-4eb6-926b-92f69cbde312" (UID: "f9778494-18c5-4eb6-926b-92f69cbde312"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:10:40.623972 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.623894    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-istio-token\" (UniqueName: \"kubernetes.io/projected/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-istio-token\") pod \"762bb1a6-9339-4c49-bf97-1e160e7b0fc6\" (UID: \"762bb1a6-9339-4c49-bf97-1e160e7b0fc6\") "
Oct 30 14:10:40.624198 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.623971    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2c83cc2-4c02-493a-948a-3a0ecf046a37-kube-api-access-rvbpc\" (UniqueName: \"kubernetes.io/projected/d2c83cc2-4c02-493a-948a-3a0ecf046a37-kube-api-access-rvbpc\") pod \"d2c83cc2-4c02-493a-948a-3a0ecf046a37\" (UID: \"d2c83cc2-4c02-493a-948a-3a0ecf046a37\") "
Oct 30 14:10:40.624198 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.624038    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2c83cc2-4c02-493a-948a-3a0ecf046a37-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2c83cc2-4c02-493a-948a-3a0ecf046a37-workload-certs\") pod \"d2c83cc2-4c02-493a-948a-3a0ecf046a37\" (UID: \"d2c83cc2-4c02-493a-948a-3a0ecf046a37\") "
Oct 30 14:10:40.624198 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.624071    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2c83cc2-4c02-493a-948a-3a0ecf046a37-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2c83cc2-4c02-493a-948a-3a0ecf046a37-istio-podinfo\") pod \"d2c83cc2-4c02-493a-948a-3a0ecf046a37\" (UID: \"d2c83cc2-4c02-493a-948a-3a0ecf046a37\") "
Oct 30 14:10:40.624198 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.624100    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2c83cc2-4c02-493a-948a-3a0ecf046a37-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2c83cc2-4c02-493a-948a-3a0ecf046a37-istiod-ca-cert\") pod \"d2c83cc2-4c02-493a-948a-3a0ecf046a37\" (UID: \"d2c83cc2-4c02-493a-948a-3a0ecf046a37\") "
Oct 30 14:10:40.624198 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.624149    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-workload-socket\") pod \"762bb1a6-9339-4c49-bf97-1e160e7b0fc6\" (UID: \"762bb1a6-9339-4c49-bf97-1e160e7b0fc6\") "
Oct 30 14:10:40.624198 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.624189    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-istio-data\") pod \"762bb1a6-9339-4c49-bf97-1e160e7b0fc6\" (UID: \"762bb1a6-9339-4c49-bf97-1e160e7b0fc6\") "
Oct 30 14:10:40.624539 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.624235    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-workload-certs\") pod \"762bb1a6-9339-4c49-bf97-1e160e7b0fc6\" (UID: \"762bb1a6-9339-4c49-bf97-1e160e7b0fc6\") "
Oct 30 14:10:40.624539 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.624296    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-istio-podinfo\") pod \"762bb1a6-9339-4c49-bf97-1e160e7b0fc6\" (UID: \"762bb1a6-9339-4c49-bf97-1e160e7b0fc6\") "
Oct 30 14:10:40.624539 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.624329    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c83cc2-4c02-493a-948a-3a0ecf046a37-workload-certs" pod "d2c83cc2-4c02-493a-948a-3a0ecf046a37" (UID: "d2c83cc2-4c02-493a-948a-3a0ecf046a37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:10:40.624539 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.624423    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-credential-socket\") pod \"762bb1a6-9339-4c49-bf97-1e160e7b0fc6\" (UID: \"762bb1a6-9339-4c49-bf97-1e160e7b0fc6\") "
Oct 30 14:10:40.624539 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.624457    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-kube-api-access-sspzw\" (UniqueName: \"kubernetes.io/projected/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-kube-api-access-sspzw\") pod \"762bb1a6-9339-4c49-bf97-1e160e7b0fc6\" (UID: \"762bb1a6-9339-4c49-bf97-1e160e7b0fc6\") "
Oct 30 14:10:40.624539 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.624497    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9778494-18c5-4eb6-926b-92f69cbde312-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9778494-18c5-4eb6-926b-92f69cbde312-istio-podinfo\") pod \"f9778494-18c5-4eb6-926b-92f69cbde312\" (UID: \"f9778494-18c5-4eb6-926b-92f69cbde312\") "
Oct 30 14:10:40.624539 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.624533    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9778494-18c5-4eb6-926b-92f69cbde312-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9778494-18c5-4eb6-926b-92f69cbde312-workload-socket\") pod \"f9778494-18c5-4eb6-926b-92f69cbde312\" (UID: \"f9778494-18c5-4eb6-926b-92f69cbde312\") "
Oct 30 14:10:40.624912 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.624555    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2c83cc2-4c02-493a-948a-3a0ecf046a37-istiod-ca-cert" pod "d2c83cc2-4c02-493a-948a-3a0ecf046a37" (UID: "d2c83cc2-4c02-493a-948a-3a0ecf046a37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:10:40.624912 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.624568    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9778494-18c5-4eb6-926b-92f69cbde312-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9778494-18c5-4eb6-926b-92f69cbde312-workload-certs\") pod \"f9778494-18c5-4eb6-926b-92f69cbde312\" (UID: \"f9778494-18c5-4eb6-926b-92f69cbde312\") "
Oct 30 14:10:40.624912 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.624624    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-istiod-ca-cert\") pod \"762bb1a6-9339-4c49-bf97-1e160e7b0fc6\" (UID: \"762bb1a6-9339-4c49-bf97-1e160e7b0fc6\") "
Oct 30 14:10:40.624912 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.624654    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2c83cc2-4c02-493a-948a-3a0ecf046a37-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2c83cc2-4c02-493a-948a-3a0ecf046a37-istio-envoy\") pod \"d2c83cc2-4c02-493a-948a-3a0ecf046a37\" (UID: \"d2c83cc2-4c02-493a-948a-3a0ecf046a37\") "
Oct 30 14:10:40.624912 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.624702    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2c83cc2-4c02-493a-948a-3a0ecf046a37-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2c83cc2-4c02-493a-948a-3a0ecf046a37-istio-data\") pod \"d2c83cc2-4c02-493a-948a-3a0ecf046a37\" (UID: \"d2c83cc2-4c02-493a-948a-3a0ecf046a37\") "
Oct 30 14:10:40.624912 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.624740    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-istio-envoy\") pod \"762bb1a6-9339-4c49-bf97-1e160e7b0fc6\" (UID: \"762bb1a6-9339-4c49-bf97-1e160e7b0fc6\") "
Oct 30 14:10:40.624912 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.624784    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9778494-18c5-4eb6-926b-92f69cbde312-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9778494-18c5-4eb6-926b-92f69cbde312-istiod-ca-cert\") pod \"f9778494-18c5-4eb6-926b-92f69cbde312\" (UID: \"f9778494-18c5-4eb6-926b-92f69cbde312\") "
Oct 30 14:10:40.624912 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.624833    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9778494-18c5-4eb6-926b-92f69cbde312-istio-token\" (UniqueName: \"kubernetes.io/projected/f9778494-18c5-4eb6-926b-92f69cbde312-istio-token\") pod \"f9778494-18c5-4eb6-926b-92f69cbde312\" (UID: \"f9778494-18c5-4eb6-926b-92f69cbde312\") "
Oct 30 14:10:40.625370 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.625214    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-istio-data" pod "762bb1a6-9339-4c49-bf97-1e160e7b0fc6" (UID: "762bb1a6-9339-4c49-bf97-1e160e7b0fc6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:10:40.625482 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.625429    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-workload-socket" pod "762bb1a6-9339-4c49-bf97-1e160e7b0fc6" (UID: "762bb1a6-9339-4c49-bf97-1e160e7b0fc6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:10:40.625546 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.625473    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9778494-18c5-4eb6-926b-92f69cbde312-workload-certs" pod "f9778494-18c5-4eb6-926b-92f69cbde312" (UID: "f9778494-18c5-4eb6-926b-92f69cbde312"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:10:40.625546 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.625483    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c83cc2-4c02-493a-948a-3a0ecf046a37-istio-data" pod "d2c83cc2-4c02-493a-948a-3a0ecf046a37" (UID: "d2c83cc2-4c02-493a-948a-3a0ecf046a37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:10:40.625760 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.625732    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-workload-certs" pod "762bb1a6-9339-4c49-bf97-1e160e7b0fc6" (UID: "762bb1a6-9339-4c49-bf97-1e160e7b0fc6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:10:40.625863 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.625843    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-istiod-ca-cert" pod "762bb1a6-9339-4c49-bf97-1e160e7b0fc6" (UID: "762bb1a6-9339-4c49-bf97-1e160e7b0fc6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:10:40.626708 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.626310    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9778494-18c5-4eb6-926b-92f69cbde312-workload-socket" pod "f9778494-18c5-4eb6-926b-92f69cbde312" (UID: "f9778494-18c5-4eb6-926b-92f69cbde312"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:10:40.626708 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.626612    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9778494-18c5-4eb6-926b-92f69cbde312-istiod-ca-cert" pod "f9778494-18c5-4eb6-926b-92f69cbde312" (UID: "f9778494-18c5-4eb6-926b-92f69cbde312"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:10:40.626708 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.626675    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-credential-socket" pod "762bb1a6-9339-4c49-bf97-1e160e7b0fc6" (UID: "762bb1a6-9339-4c49-bf97-1e160e7b0fc6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:10:40.627374 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.627351    5093 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/926c412f-623c-489d-8100-c66c75a5f478-workload-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:40.627514 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.627498    5093 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9778494-18c5-4eb6-926b-92f69cbde312-credential-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:40.627616 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.627602    5093 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c83cc2-4c02-493a-948a-3a0ecf046a37-workload-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:40.627711 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.627697    5093 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c83cc2-4c02-493a-948a-3a0ecf046a37-credential-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:40.627836 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.627822    5093 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9778494-18c5-4eb6-926b-92f69cbde312-istio-data\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:40.627954 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.627937    5093 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/926c412f-623c-489d-8100-c66c75a5f478-credential-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:40.628050 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.628037    5093 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2c83cc2-4c02-493a-948a-3a0ecf046a37-workload-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:40.628273 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.628258    5093 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2c83cc2-4c02-493a-948a-3a0ecf046a37-istiod-ca-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:40.628389 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.628368    5093 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-workload-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:40.628389 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.628388    5093 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-istio-data\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:40.628595 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.628404    5093 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-workload-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:40.628595 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.628418    5093 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/926c412f-623c-489d-8100-c66c75a5f478-istiod-ca-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:40.628595 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.628431    5093 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/926c412f-623c-489d-8100-c66c75a5f478-istio-podinfo\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:40.628595 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.628443    5093 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-credential-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:40.628595 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.628457    5093 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/926c412f-623c-489d-8100-c66c75a5f478-workload-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:40.628595 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.628472    5093 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/926c412f-623c-489d-8100-c66c75a5f478-istio-data\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:40.628595 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.628485    5093 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9778494-18c5-4eb6-926b-92f69cbde312-workload-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:40.628595 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.628498    5093 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9778494-18c5-4eb6-926b-92f69cbde312-workload-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:40.628595 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.628514    5093 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-istiod-ca-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:40.628595 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.628564    5093 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2c83cc2-4c02-493a-948a-3a0ecf046a37-istio-data\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:40.628595 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.628577    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djmrs\" (UniqueName: \"kubernetes.io/projected/926c412f-623c-489d-8100-c66c75a5f478-kube-api-access-djmrs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:40.628595 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.628593    5093 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/926c412f-623c-489d-8100-c66c75a5f478-istio-envoy\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:40.629310 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.628606    5093 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9778494-18c5-4eb6-926b-92f69cbde312-istiod-ca-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:40.629310 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.628617    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/926c412f-623c-489d-8100-c66c75a5f478-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:40.629310 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.629100    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2c83cc2-4c02-493a-948a-3a0ecf046a37-istio-podinfo" pod "d2c83cc2-4c02-493a-948a-3a0ecf046a37" (UID: "d2c83cc2-4c02-493a-948a-3a0ecf046a37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:10:40.630682 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.630649    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9778494-18c5-4eb6-926b-92f69cbde312-istio-token" pod "f9778494-18c5-4eb6-926b-92f69cbde312" (UID: "f9778494-18c5-4eb6-926b-92f69cbde312"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:10:40.631575 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.631518    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-istio-podinfo" pod "762bb1a6-9339-4c49-bf97-1e160e7b0fc6" (UID: "762bb1a6-9339-4c49-bf97-1e160e7b0fc6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:10:40.631575 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.631540    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c83cc2-4c02-493a-948a-3a0ecf046a37-istio-envoy" pod "d2c83cc2-4c02-493a-948a-3a0ecf046a37" (UID: "d2c83cc2-4c02-493a-948a-3a0ecf046a37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:10:40.631766 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.631583    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-kube-api-access-sspzw" pod "762bb1a6-9339-4c49-bf97-1e160e7b0fc6" (UID: "762bb1a6-9339-4c49-bf97-1e160e7b0fc6"). InnerVolumeSpecName "kube-api-access-sspzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:10:40.631766 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.631658    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2c83cc2-4c02-493a-948a-3a0ecf046a37-kube-api-access-rvbpc" pod "d2c83cc2-4c02-493a-948a-3a0ecf046a37" (UID: "d2c83cc2-4c02-493a-948a-3a0ecf046a37"). InnerVolumeSpecName "kube-api-access-rvbpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:10:40.631766 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.631686    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9778494-18c5-4eb6-926b-92f69cbde312-istio-envoy" pod "f9778494-18c5-4eb6-926b-92f69cbde312" (UID: "f9778494-18c5-4eb6-926b-92f69cbde312"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:10:40.631766 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.631718    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9778494-18c5-4eb6-926b-92f69cbde312-kube-api-access-cnpl7" pod "f9778494-18c5-4eb6-926b-92f69cbde312" (UID: "f9778494-18c5-4eb6-926b-92f69cbde312"). InnerVolumeSpecName "kube-api-access-cnpl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:10:40.631935 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.631774    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-istio-token" pod "762bb1a6-9339-4c49-bf97-1e160e7b0fc6" (UID: "762bb1a6-9339-4c49-bf97-1e160e7b0fc6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:10:40.631935 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.631776    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2c83cc2-4c02-493a-948a-3a0ecf046a37-istio-token" pod "d2c83cc2-4c02-493a-948a-3a0ecf046a37" (UID: "d2c83cc2-4c02-493a-948a-3a0ecf046a37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:10:40.632514 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.632484    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9778494-18c5-4eb6-926b-92f69cbde312-istio-podinfo" pod "f9778494-18c5-4eb6-926b-92f69cbde312" (UID: "f9778494-18c5-4eb6-926b-92f69cbde312"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:10:40.632993 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.632976    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-istio-envoy" pod "762bb1a6-9339-4c49-bf97-1e160e7b0fc6" (UID: "762bb1a6-9339-4c49-bf97-1e160e7b0fc6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:10:40.729943 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.729901    5093 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-istio-podinfo\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:40.729943 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.729942    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sspzw\" (UniqueName: \"kubernetes.io/projected/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-kube-api-access-sspzw\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:40.730199 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.729953    5093 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9778494-18c5-4eb6-926b-92f69cbde312-istio-podinfo\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:40.730199 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.729962    5093 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2c83cc2-4c02-493a-948a-3a0ecf046a37-istio-envoy\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:40.730199 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.729970    5093 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-istio-envoy\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:40.730199 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.729978    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9778494-18c5-4eb6-926b-92f69cbde312-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:40.730199 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.729987    5093 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9778494-18c5-4eb6-926b-92f69cbde312-istio-envoy\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:40.730199 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.729995    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnpl7\" (UniqueName: \"kubernetes.io/projected/f9778494-18c5-4eb6-926b-92f69cbde312-kube-api-access-cnpl7\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:40.730199 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.730002    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2c83cc2-4c02-493a-948a-3a0ecf046a37-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:40.730199 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.730014    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/762bb1a6-9339-4c49-bf97-1e160e7b0fc6-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:40.730199 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.730032    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvbpc\" (UniqueName: \"kubernetes.io/projected/d2c83cc2-4c02-493a-948a-3a0ecf046a37-kube-api-access-rvbpc\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:40.730199 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:40.730044    5093 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2c83cc2-4c02-493a-948a-3a0ecf046a37-istio-podinfo\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:41.000375 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:41.000340    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a51f3807-9699-4d02-8839-302a7f44afa3" path="/var/lib/kubelet/pods/a51f3807-9699-4d02-8839-302a7f44afa3/volumes"
Oct 30 14:10:41.009753 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:10:41.009704    5093 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7686-4ba6d9\" is forbidden: User \"system:node:ip-10-0-134-31.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-31.ec2.internal' and this object" podUID="a51f3807-9699-4d02-8839-302a7f44afa3" pod="opendatahub/test-curl-7686-4ba6d9"
Oct 30 14:10:41.085494 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:41.085455    5093 generic.go:356] "Generic (PLEG): container finished" podID="f9778494-18c5-4eb6-926b-92f69cbde312" containerID="36d337dfb735313c530f1c21e21c4e270679971d466c5794a3eb797d63bb2360" exitCode=0
Oct 30 14:10:41.085638 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:41.085519    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5"
Oct 30 14:10:41.085638 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:41.085533    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5" event={"ID":"f9778494-18c5-4eb6-926b-92f69cbde312","Type":"ContainerDied","Data":"36d337dfb735313c530f1c21e21c4e270679971d466c5794a3eb797d63bb2360"}
Oct 30 14:10:41.085638 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:41.085594    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5" event={"ID":"f9778494-18c5-4eb6-926b-92f69cbde312","Type":"ContainerDied","Data":"6efb52b9c451bca3ac0545fca41c2f531d33ecf1e44e5d30af3349485f7cf690"}
Oct 30 14:10:41.085638 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:41.085629    5093 scope.go:122] "RemoveContainer" containerID="36d337dfb735313c530f1c21e21c4e270679971d466c5794a3eb797d63bb2360"
Oct 30 14:10:41.087321 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:41.087285    5093 generic.go:356] "Generic (PLEG): container finished" podID="926c412f-623c-489d-8100-c66c75a5f478" containerID="1b59835711224dbaf4410d797ab735072aa6903fe877d6734184f207c8ed1bc8" exitCode=0
Oct 30 14:10:41.087438 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:41.087346    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2"
Oct 30 14:10:41.087438 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:41.087394    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2" event={"ID":"926c412f-623c-489d-8100-c66c75a5f478","Type":"ContainerDied","Data":"1b59835711224dbaf4410d797ab735072aa6903fe877d6734184f207c8ed1bc8"}
Oct 30 14:10:41.087565 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:41.087462    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2" event={"ID":"926c412f-623c-489d-8100-c66c75a5f478","Type":"ContainerDied","Data":"af782d5b6f1423bf1ad7f01e532745561de650a8708900ec5b4fc85ed841e8d9"}
Oct 30 14:10:41.089566 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:41.089544    5093 generic.go:356] "Generic (PLEG): container finished" podID="762bb1a6-9339-4c49-bf97-1e160e7b0fc6" containerID="b3e545a30c051f6f4a59c8ea60a716fc6d246eac9915f0c924fcb1ac06cb46bb" exitCode=0
Oct 30 14:10:41.089691 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:41.089604    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b"
Oct 30 14:10:41.089691 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:41.089629    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b" event={"ID":"762bb1a6-9339-4c49-bf97-1e160e7b0fc6","Type":"ContainerDied","Data":"b3e545a30c051f6f4a59c8ea60a716fc6d246eac9915f0c924fcb1ac06cb46bb"}
Oct 30 14:10:41.089812 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:41.089696    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b" event={"ID":"762bb1a6-9339-4c49-bf97-1e160e7b0fc6","Type":"ContainerDied","Data":"2c7c5fdaa090b87fe1938896a70a96d62e189a9035222507213e353240a276e0"}
Oct 30 14:10:41.092757 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:41.092735    5093 generic.go:356] "Generic (PLEG): container finished" podID="d2c83cc2-4c02-493a-948a-3a0ecf046a37" containerID="8234e542f8c3186d92daa8ef0662dbea057aa22f4553b94d48752bdc7dddd7cb" exitCode=0
Oct 30 14:10:41.092877 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:41.092772    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2" event={"ID":"d2c83cc2-4c02-493a-948a-3a0ecf046a37","Type":"ContainerDied","Data":"8234e542f8c3186d92daa8ef0662dbea057aa22f4553b94d48752bdc7dddd7cb"}
Oct 30 14:10:41.092877 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:41.092797    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2"
Oct 30 14:10:41.092877 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:41.092811    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2" event={"ID":"d2c83cc2-4c02-493a-948a-3a0ecf046a37","Type":"ContainerDied","Data":"0c941e74875452cea73df0b6d815603f6defac529c32b13a804caba000354e34"}
Oct 30 14:10:41.106011 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:41.105984    5093 scope.go:122] "RemoveContainer" containerID="36d337dfb735313c530f1c21e21c4e270679971d466c5794a3eb797d63bb2360"
Oct 30 14:10:41.106409 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:10:41.106378    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36d337dfb735313c530f1c21e21c4e270679971d466c5794a3eb797d63bb2360\": container with ID starting with 36d337dfb735313c530f1c21e21c4e270679971d466c5794a3eb797d63bb2360 not found: ID does not exist" containerID="36d337dfb735313c530f1c21e21c4e270679971d466c5794a3eb797d63bb2360"
Oct 30 14:10:41.106552 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:41.106412    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36d337dfb735313c530f1c21e21c4e270679971d466c5794a3eb797d63bb2360"} err="failed to get container status \"36d337dfb735313c530f1c21e21c4e270679971d466c5794a3eb797d63bb2360\": rpc error: code = NotFound desc = could not find container \"36d337dfb735313c530f1c21e21c4e270679971d466c5794a3eb797d63bb2360\": container with ID starting with 36d337dfb735313c530f1c21e21c4e270679971d466c5794a3eb797d63bb2360 not found: ID does not exist"
Oct 30 14:10:41.106552 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:41.106431    5093 scope.go:122] "RemoveContainer" containerID="1b59835711224dbaf4410d797ab735072aa6903fe877d6734184f207c8ed1bc8"
Oct 30 14:10:41.116713 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:41.116693    5093 scope.go:122] "RemoveContainer" containerID="1b59835711224dbaf4410d797ab735072aa6903fe877d6734184f207c8ed1bc8"
Oct 30 14:10:41.117050 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:10:41.117027    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b59835711224dbaf4410d797ab735072aa6903fe877d6734184f207c8ed1bc8\": container with ID starting with 1b59835711224dbaf4410d797ab735072aa6903fe877d6734184f207c8ed1bc8 not found: ID does not exist" containerID="1b59835711224dbaf4410d797ab735072aa6903fe877d6734184f207c8ed1bc8"
Oct 30 14:10:41.117185 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:41.117056    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b59835711224dbaf4410d797ab735072aa6903fe877d6734184f207c8ed1bc8"} err="failed to get container status \"1b59835711224dbaf4410d797ab735072aa6903fe877d6734184f207c8ed1bc8\": rpc error: code = NotFound desc = could not find container \"1b59835711224dbaf4410d797ab735072aa6903fe877d6734184f207c8ed1bc8\": container with ID starting with 1b59835711224dbaf4410d797ab735072aa6903fe877d6734184f207c8ed1bc8 not found: ID does not exist"
Oct 30 14:10:41.117185 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:41.117075    5093 scope.go:122] "RemoveContainer" containerID="b3e545a30c051f6f4a59c8ea60a716fc6d246eac9915f0c924fcb1ac06cb46bb"
Oct 30 14:10:41.127162 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:41.127112    5093 scope.go:122] "RemoveContainer" containerID="b3e545a30c051f6f4a59c8ea60a716fc6d246eac9915f0c924fcb1ac06cb46bb"
Oct 30 14:10:41.127455 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:10:41.127438    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3e545a30c051f6f4a59c8ea60a716fc6d246eac9915f0c924fcb1ac06cb46bb\": container with ID starting with b3e545a30c051f6f4a59c8ea60a716fc6d246eac9915f0c924fcb1ac06cb46bb not found: ID does not exist" containerID="b3e545a30c051f6f4a59c8ea60a716fc6d246eac9915f0c924fcb1ac06cb46bb"
Oct 30 14:10:41.127521 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:41.127462    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3e545a30c051f6f4a59c8ea60a716fc6d246eac9915f0c924fcb1ac06cb46bb"} err="failed to get container status \"b3e545a30c051f6f4a59c8ea60a716fc6d246eac9915f0c924fcb1ac06cb46bb\": rpc error: code = NotFound desc = could not find container \"b3e545a30c051f6f4a59c8ea60a716fc6d246eac9915f0c924fcb1ac06cb46bb\": container with ID starting with b3e545a30c051f6f4a59c8ea60a716fc6d246eac9915f0c924fcb1ac06cb46bb not found: ID does not exist"
Oct 30 14:10:41.127521 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:41.127485    5093 scope.go:122] "RemoveContainer" containerID="8234e542f8c3186d92daa8ef0662dbea057aa22f4553b94d48752bdc7dddd7cb"
Oct 30 14:10:41.137908 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:41.137887    5093 scope.go:122] "RemoveContainer" containerID="8234e542f8c3186d92daa8ef0662dbea057aa22f4553b94d48752bdc7dddd7cb"
Oct 30 14:10:41.138273 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:10:41.138249    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8234e542f8c3186d92daa8ef0662dbea057aa22f4553b94d48752bdc7dddd7cb\": container with ID starting with 8234e542f8c3186d92daa8ef0662dbea057aa22f4553b94d48752bdc7dddd7cb not found: ID does not exist" containerID="8234e542f8c3186d92daa8ef0662dbea057aa22f4553b94d48752bdc7dddd7cb"
Oct 30 14:10:41.138349 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:41.138284    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8234e542f8c3186d92daa8ef0662dbea057aa22f4553b94d48752bdc7dddd7cb"} err="failed to get container status \"8234e542f8c3186d92daa8ef0662dbea057aa22f4553b94d48752bdc7dddd7cb\": rpc error: code = NotFound desc = could not find container \"8234e542f8c3186d92daa8ef0662dbea057aa22f4553b94d48752bdc7dddd7cb\": container with ID starting with 8234e542f8c3186d92daa8ef0662dbea057aa22f4553b94d48752bdc7dddd7cb not found: ID does not exist"
Oct 30 14:10:41.187739 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:41.187707    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2"]
Oct 30 14:10:41.200620 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:41.200582    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5lnv2"]
Oct 30 14:10:41.226844 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:41.226798    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b"]
Oct 30 14:10:41.229715 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:41.229686    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2k8b"]
Oct 30 14:10:41.264313 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:41.264218    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5"]
Oct 30 14:10:41.270410 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:41.270378    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b6ph5"]
Oct 30 14:10:41.308493 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:41.308457    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2"]
Oct 30 14:10:41.311677 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:41.311641    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwhx2"]
Oct 30 14:10:43.000494 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:43.000453    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="762bb1a6-9339-4c49-bf97-1e160e7b0fc6" path="/var/lib/kubelet/pods/762bb1a6-9339-4c49-bf97-1e160e7b0fc6/volumes"
Oct 30 14:10:43.001025 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:43.000951    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="926c412f-623c-489d-8100-c66c75a5f478" path="/var/lib/kubelet/pods/926c412f-623c-489d-8100-c66c75a5f478/volumes"
Oct 30 14:10:43.001410 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:43.001391    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2c83cc2-4c02-493a-948a-3a0ecf046a37" path="/var/lib/kubelet/pods/d2c83cc2-4c02-493a-948a-3a0ecf046a37/volumes"
Oct 30 14:10:43.002209 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:43.002191    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9778494-18c5-4eb6-926b-92f69cbde312" path="/var/lib/kubelet/pods/f9778494-18c5-4eb6-926b-92f69cbde312/volumes"
Oct 30 14:10:49.615118 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:49.615076    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm"]
Oct 30 14:10:49.615656 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:49.615318    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm" podUID="b0d3d9d3-593a-478d-963b-d8b4b395a661" containerName="istio-proxy" containerID="cri-o://4e34bdd9f7c4a57d1a0b007b163c1c302f5100fbb4884adc7c1c3b66a32afe10" gracePeriod=30
Oct 30 14:10:51.167161 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:10:51.167090    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 14:10:54.868157 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:54.867181    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-f7ee8a/rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn"]
Oct 30 14:10:54.906328 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:54.906300    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm"
Oct 30 14:10:54.996888 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:54.996857    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0d3d9d3-593a-478d-963b-d8b4b395a661-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0d3d9d3-593a-478d-963b-d8b4b395a661-istio-podinfo\") pod \"b0d3d9d3-593a-478d-963b-d8b4b395a661\" (UID: \"b0d3d9d3-593a-478d-963b-d8b4b395a661\") "
Oct 30 14:10:54.997070 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:54.996896    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0d3d9d3-593a-478d-963b-d8b4b395a661-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0d3d9d3-593a-478d-963b-d8b4b395a661-istio-envoy\") pod \"b0d3d9d3-593a-478d-963b-d8b4b395a661\" (UID: \"b0d3d9d3-593a-478d-963b-d8b4b395a661\") "
Oct 30 14:10:54.997070 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:54.996940    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0d3d9d3-593a-478d-963b-d8b4b395a661-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0d3d9d3-593a-478d-963b-d8b4b395a661-istio-data\") pod \"b0d3d9d3-593a-478d-963b-d8b4b395a661\" (UID: \"b0d3d9d3-593a-478d-963b-d8b4b395a661\") "
Oct 30 14:10:54.997070 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:54.997024    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0d3d9d3-593a-478d-963b-d8b4b395a661-istio-token\" (UniqueName: \"kubernetes.io/projected/b0d3d9d3-593a-478d-963b-d8b4b395a661-istio-token\") pod \"b0d3d9d3-593a-478d-963b-d8b4b395a661\" (UID: \"b0d3d9d3-593a-478d-963b-d8b4b395a661\") "
Oct 30 14:10:54.997286 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:54.997240    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d3d9d3-593a-478d-963b-d8b4b395a661-istio-data" pod "b0d3d9d3-593a-478d-963b-d8b4b395a661" (UID: "b0d3d9d3-593a-478d-963b-d8b4b395a661"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:10:54.997518 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:54.997221    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0d3d9d3-593a-478d-963b-d8b4b395a661-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0d3d9d3-593a-478d-963b-d8b4b395a661-workload-socket\") pod \"b0d3d9d3-593a-478d-963b-d8b4b395a661\" (UID: \"b0d3d9d3-593a-478d-963b-d8b4b395a661\") "
Oct 30 14:10:54.997596 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:54.997574    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0d3d9d3-593a-478d-963b-d8b4b395a661-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0d3d9d3-593a-478d-963b-d8b4b395a661-workload-certs\") pod \"b0d3d9d3-593a-478d-963b-d8b4b395a661\" (UID: \"b0d3d9d3-593a-478d-963b-d8b4b395a661\") "
Oct 30 14:10:54.997596 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:54.997487    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d3d9d3-593a-478d-963b-d8b4b395a661-workload-socket" pod "b0d3d9d3-593a-478d-963b-d8b4b395a661" (UID: "b0d3d9d3-593a-478d-963b-d8b4b395a661"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:10:54.997687 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:54.997639    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0d3d9d3-593a-478d-963b-d8b4b395a661-kube-api-access-nkkt7\" (UniqueName: \"kubernetes.io/projected/b0d3d9d3-593a-478d-963b-d8b4b395a661-kube-api-access-nkkt7\") pod \"b0d3d9d3-593a-478d-963b-d8b4b395a661\" (UID: \"b0d3d9d3-593a-478d-963b-d8b4b395a661\") "
Oct 30 14:10:54.997750 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:54.997687    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0d3d9d3-593a-478d-963b-d8b4b395a661-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0d3d9d3-593a-478d-963b-d8b4b395a661-istiod-ca-cert\") pod \"b0d3d9d3-593a-478d-963b-d8b4b395a661\" (UID: \"b0d3d9d3-593a-478d-963b-d8b4b395a661\") "
Oct 30 14:10:54.997860 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:54.997841    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0d3d9d3-593a-478d-963b-d8b4b395a661-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0d3d9d3-593a-478d-963b-d8b4b395a661-credential-socket\") pod \"b0d3d9d3-593a-478d-963b-d8b4b395a661\" (UID: \"b0d3d9d3-593a-478d-963b-d8b4b395a661\") "
Oct 30 14:10:54.998173 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:54.998145    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d3d9d3-593a-478d-963b-d8b4b395a661-workload-certs" pod "b0d3d9d3-593a-478d-963b-d8b4b395a661" (UID: "b0d3d9d3-593a-478d-963b-d8b4b395a661"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:10:54.998398 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:54.998370    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d3d9d3-593a-478d-963b-d8b4b395a661-credential-socket" pod "b0d3d9d3-593a-478d-963b-d8b4b395a661" (UID: "b0d3d9d3-593a-478d-963b-d8b4b395a661"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:10:54.998519 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:54.998494    5093 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0d3d9d3-593a-478d-963b-d8b4b395a661-workload-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:54.998586 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:54.998527    5093 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0d3d9d3-593a-478d-963b-d8b4b395a661-workload-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:54.998586 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:54.998542    5093 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0d3d9d3-593a-478d-963b-d8b4b395a661-istio-data\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:54.998686 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:54.998660    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0d3d9d3-593a-478d-963b-d8b4b395a661-istiod-ca-cert" pod "b0d3d9d3-593a-478d-963b-d8b4b395a661" (UID: "b0d3d9d3-593a-478d-963b-d8b4b395a661"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:10:54.999838 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:54.999811    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0d3d9d3-593a-478d-963b-d8b4b395a661-istio-podinfo" pod "b0d3d9d3-593a-478d-963b-d8b4b395a661" (UID: "b0d3d9d3-593a-478d-963b-d8b4b395a661"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:10:55.009701 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:55.009665    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d3d9d3-593a-478d-963b-d8b4b395a661-istio-envoy" pod "b0d3d9d3-593a-478d-963b-d8b4b395a661" (UID: "b0d3d9d3-593a-478d-963b-d8b4b395a661"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:10:55.009701 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:55.009695    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0d3d9d3-593a-478d-963b-d8b4b395a661-istio-token" pod "b0d3d9d3-593a-478d-963b-d8b4b395a661" (UID: "b0d3d9d3-593a-478d-963b-d8b4b395a661"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:10:55.012098 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:55.012068    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0d3d9d3-593a-478d-963b-d8b4b395a661-kube-api-access-nkkt7" pod "b0d3d9d3-593a-478d-963b-d8b4b395a661" (UID: "b0d3d9d3-593a-478d-963b-d8b4b395a661"). InnerVolumeSpecName "kube-api-access-nkkt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:10:55.100409 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:55.100359    5093 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0d3d9d3-593a-478d-963b-d8b4b395a661-credential-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:55.100409 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:55.100393    5093 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0d3d9d3-593a-478d-963b-d8b4b395a661-istio-podinfo\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:55.100409 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:55.100408    5093 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0d3d9d3-593a-478d-963b-d8b4b395a661-istio-envoy\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:55.100409 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:55.100420    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0d3d9d3-593a-478d-963b-d8b4b395a661-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:55.100782 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:55.100433    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkkt7\" (UniqueName: \"kubernetes.io/projected/b0d3d9d3-593a-478d-963b-d8b4b395a661-kube-api-access-nkkt7\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:55.100782 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:55.100447    5093 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0d3d9d3-593a-478d-963b-d8b4b395a661-istiod-ca-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:10:55.196704 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:55.196667    5093 generic.go:356] "Generic (PLEG): container finished" podID="b0d3d9d3-593a-478d-963b-d8b4b395a661" containerID="4e34bdd9f7c4a57d1a0b007b163c1c302f5100fbb4884adc7c1c3b66a32afe10" exitCode=0
Oct 30 14:10:55.196870 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:55.196724    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm" event={"ID":"b0d3d9d3-593a-478d-963b-d8b4b395a661","Type":"ContainerDied","Data":"4e34bdd9f7c4a57d1a0b007b163c1c302f5100fbb4884adc7c1c3b66a32afe10"}
Oct 30 14:10:55.196870 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:55.196727    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm"
Oct 30 14:10:55.196870 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:55.196765    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm" event={"ID":"b0d3d9d3-593a-478d-963b-d8b4b395a661","Type":"ContainerDied","Data":"e488a8f5502f02bab02f075053473ed243c5c8ecf184cfdaa4ff0ec7a3e85e64"}
Oct 30 14:10:55.196870 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:55.196788    5093 scope.go:122] "RemoveContainer" containerID="4e34bdd9f7c4a57d1a0b007b163c1c302f5100fbb4884adc7c1c3b66a32afe10"
Oct 30 14:10:55.216091 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:55.216067    5093 scope.go:122] "RemoveContainer" containerID="b3b733ee51a45db343a28e5099a1558c7507df79b6084d95825f8c769dc461d3"
Oct 30 14:10:55.229510 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:55.229470    5093 scope.go:122] "RemoveContainer" containerID="4e34bdd9f7c4a57d1a0b007b163c1c302f5100fbb4884adc7c1c3b66a32afe10"
Oct 30 14:10:55.229864 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:10:55.229840    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e34bdd9f7c4a57d1a0b007b163c1c302f5100fbb4884adc7c1c3b66a32afe10\": container with ID starting with 4e34bdd9f7c4a57d1a0b007b163c1c302f5100fbb4884adc7c1c3b66a32afe10 not found: ID does not exist" containerID="4e34bdd9f7c4a57d1a0b007b163c1c302f5100fbb4884adc7c1c3b66a32afe10"
Oct 30 14:10:55.229961 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:55.229877    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e34bdd9f7c4a57d1a0b007b163c1c302f5100fbb4884adc7c1c3b66a32afe10"} err="failed to get container status \"4e34bdd9f7c4a57d1a0b007b163c1c302f5100fbb4884adc7c1c3b66a32afe10\": rpc error: code = NotFound desc = could not find container \"4e34bdd9f7c4a57d1a0b007b163c1c302f5100fbb4884adc7c1c3b66a32afe10\": container with ID starting with 4e34bdd9f7c4a57d1a0b007b163c1c302f5100fbb4884adc7c1c3b66a32afe10 not found: ID does not exist"
Oct 30 14:10:55.229961 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:55.229898    5093 scope.go:122] "RemoveContainer" containerID="b3b733ee51a45db343a28e5099a1558c7507df79b6084d95825f8c769dc461d3"
Oct 30 14:10:55.230199 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:10:55.230179    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3b733ee51a45db343a28e5099a1558c7507df79b6084d95825f8c769dc461d3\": container with ID starting with b3b733ee51a45db343a28e5099a1558c7507df79b6084d95825f8c769dc461d3 not found: ID does not exist" containerID="b3b733ee51a45db343a28e5099a1558c7507df79b6084d95825f8c769dc461d3"
Oct 30 14:10:55.230257 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:55.230206    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3b733ee51a45db343a28e5099a1558c7507df79b6084d95825f8c769dc461d3"} err="failed to get container status \"b3b733ee51a45db343a28e5099a1558c7507df79b6084d95825f8c769dc461d3\": rpc error: code = NotFound desc = could not find container \"b3b733ee51a45db343a28e5099a1558c7507df79b6084d95825f8c769dc461d3\": container with ID starting with b3b733ee51a45db343a28e5099a1558c7507df79b6084d95825f8c769dc461d3 not found: ID does not exist"
Oct 30 14:10:55.259285 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:55.259248    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm"]
Oct 30 14:10:55.272044 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:55.272003    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-74589c-openshift-default-5c97c5c997-q7bwm"]
Oct 30 14:10:56.999702 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:10:56.999666    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0d3d9d3-593a-478d-963b-d8b4b395a661" path="/var/lib/kubelet/pods/b0d3d9d3-593a-478d-963b-d8b4b395a661/volumes"
Oct 30 14:11:07.211863 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:07.211820    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c3efb2-5f44b7cbcf-4zkz8"]
Oct 30 14:11:09.903213 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:09.903118    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-f7ee8a/rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn" podUID="991b571e-8f93-47b0-9bb8-ab71ee26897c" containerName="main" containerID="cri-o://c2062ed9ec3ecb88cc205985959891486b1d3dc8d242bcb15c8ee0b465e51479" gracePeriod=45
Oct 30 14:11:09.998356 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:09.998320    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c3efb2-6d4548b87-n444z"]
Oct 30 14:11:09.998586 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:09.998555    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c3efb2-6d4548b87-n444z" podUID="6b42343a-61e7-468a-aae9-701026a365bc" containerName="maas-api" containerID="cri-o://63843bd2872be329fb9394fdc18a1b39682ce01d73eea1d94f325d0870b96669" gracePeriod=30
Oct 30 14:11:10.232426 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:10.232382    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-f7ee8a/rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn"
Oct 30 14:11:10.289952 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:10.289899    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/991b571e-8f93-47b0-9bb8-ab71ee26897c-kube-api-access-vgpws\" (UniqueName: \"kubernetes.io/projected/991b571e-8f93-47b0-9bb8-ab71ee26897c-kube-api-access-vgpws\") pod \"991b571e-8f93-47b0-9bb8-ab71ee26897c\" (UID: \"991b571e-8f93-47b0-9bb8-ab71ee26897c\") "
Oct 30 14:11:10.290190 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:10.289977    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/991b571e-8f93-47b0-9bb8-ab71ee26897c-tls-certs\" (UniqueName: \"kubernetes.io/secret/991b571e-8f93-47b0-9bb8-ab71ee26897c-tls-certs\") pod \"991b571e-8f93-47b0-9bb8-ab71ee26897c\" (UID: \"991b571e-8f93-47b0-9bb8-ab71ee26897c\") "
Oct 30 14:11:10.290190 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:10.290100    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/991b571e-8f93-47b0-9bb8-ab71ee26897c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/991b571e-8f93-47b0-9bb8-ab71ee26897c-model-cache\") pod \"991b571e-8f93-47b0-9bb8-ab71ee26897c\" (UID: \"991b571e-8f93-47b0-9bb8-ab71ee26897c\") "
Oct 30 14:11:10.290190 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:10.290173    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/991b571e-8f93-47b0-9bb8-ab71ee26897c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/991b571e-8f93-47b0-9bb8-ab71ee26897c-dshm\") pod \"991b571e-8f93-47b0-9bb8-ab71ee26897c\" (UID: \"991b571e-8f93-47b0-9bb8-ab71ee26897c\") "
Oct 30 14:11:10.290396 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:10.290221    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/991b571e-8f93-47b0-9bb8-ab71ee26897c-home\" (UniqueName: \"kubernetes.io/empty-dir/991b571e-8f93-47b0-9bb8-ab71ee26897c-home\") pod \"991b571e-8f93-47b0-9bb8-ab71ee26897c\" (UID: \"991b571e-8f93-47b0-9bb8-ab71ee26897c\") "
Oct 30 14:11:10.290396 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:10.290253    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/991b571e-8f93-47b0-9bb8-ab71ee26897c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/991b571e-8f93-47b0-9bb8-ab71ee26897c-tmp-dir\") pod \"991b571e-8f93-47b0-9bb8-ab71ee26897c\" (UID: \"991b571e-8f93-47b0-9bb8-ab71ee26897c\") "
Oct 30 14:11:10.290926 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:10.290903    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/991b571e-8f93-47b0-9bb8-ab71ee26897c-tmp-dir" pod "991b571e-8f93-47b0-9bb8-ab71ee26897c" (UID: "991b571e-8f93-47b0-9bb8-ab71ee26897c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:11:10.294890 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:10.294855    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/991b571e-8f93-47b0-9bb8-ab71ee26897c-kube-api-access-vgpws" pod "991b571e-8f93-47b0-9bb8-ab71ee26897c" (UID: "991b571e-8f93-47b0-9bb8-ab71ee26897c"). InnerVolumeSpecName "kube-api-access-vgpws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:11:10.295252 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:10.295228    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/991b571e-8f93-47b0-9bb8-ab71ee26897c-home" pod "991b571e-8f93-47b0-9bb8-ab71ee26897c" (UID: "991b571e-8f93-47b0-9bb8-ab71ee26897c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:11:10.295351 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:10.295182    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/991b571e-8f93-47b0-9bb8-ab71ee26897c-model-cache" pod "991b571e-8f93-47b0-9bb8-ab71ee26897c" (UID: "991b571e-8f93-47b0-9bb8-ab71ee26897c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:11:10.310447 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:10.303427    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/991b571e-8f93-47b0-9bb8-ab71ee26897c-dshm" pod "991b571e-8f93-47b0-9bb8-ab71ee26897c" (UID: "991b571e-8f93-47b0-9bb8-ab71ee26897c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:11:10.310884 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:10.310598    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/991b571e-8f93-47b0-9bb8-ab71ee26897c-tls-certs" pod "991b571e-8f93-47b0-9bb8-ab71ee26897c" (UID: "991b571e-8f93-47b0-9bb8-ab71ee26897c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:11:10.314588 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:10.314551    5093 generic.go:356] "Generic (PLEG): container finished" podID="6b42343a-61e7-468a-aae9-701026a365bc" containerID="63843bd2872be329fb9394fdc18a1b39682ce01d73eea1d94f325d0870b96669" exitCode=0
Oct 30 14:11:10.314733 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:10.314628    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c3efb2-6d4548b87-n444z" event={"ID":"6b42343a-61e7-468a-aae9-701026a365bc","Type":"ContainerDied","Data":"63843bd2872be329fb9394fdc18a1b39682ce01d73eea1d94f325d0870b96669"}
Oct 30 14:11:10.316491 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:10.316468    5093 generic.go:356] "Generic (PLEG): container finished" podID="991b571e-8f93-47b0-9bb8-ab71ee26897c" containerID="c2062ed9ec3ecb88cc205985959891486b1d3dc8d242bcb15c8ee0b465e51479" exitCode=0
Oct 30 14:11:10.316618 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:10.316514    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-f7ee8a/rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn" event={"ID":"991b571e-8f93-47b0-9bb8-ab71ee26897c","Type":"ContainerDied","Data":"c2062ed9ec3ecb88cc205985959891486b1d3dc8d242bcb15c8ee0b465e51479"}
Oct 30 14:11:10.316618 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:10.316537    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-f7ee8a/rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn"
Oct 30 14:11:10.316618 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:10.316553    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-f7ee8a/rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn" event={"ID":"991b571e-8f93-47b0-9bb8-ab71ee26897c","Type":"ContainerDied","Data":"0c927295d96d5a35d0d55123d804b3a01ae19cd8529dc9fcd195b1df13ec0cd8"}
Oct 30 14:11:10.316618 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:10.316578    5093 scope.go:122] "RemoveContainer" containerID="c2062ed9ec3ecb88cc205985959891486b1d3dc8d242bcb15c8ee0b465e51479"
Oct 30 14:11:10.326732 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:10.326702    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c3efb2-6d4548b87-n444z"
Oct 30 14:11:10.330696 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:10.329149    5093 scope.go:122] "RemoveContainer" containerID="c2062ed9ec3ecb88cc205985959891486b1d3dc8d242bcb15c8ee0b465e51479"
Oct 30 14:11:10.330696 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:11:10.329520    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2062ed9ec3ecb88cc205985959891486b1d3dc8d242bcb15c8ee0b465e51479\": container with ID starting with c2062ed9ec3ecb88cc205985959891486b1d3dc8d242bcb15c8ee0b465e51479 not found: ID does not exist" containerID="c2062ed9ec3ecb88cc205985959891486b1d3dc8d242bcb15c8ee0b465e51479"
Oct 30 14:11:10.330696 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:10.329564    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2062ed9ec3ecb88cc205985959891486b1d3dc8d242bcb15c8ee0b465e51479"} err="failed to get container status \"c2062ed9ec3ecb88cc205985959891486b1d3dc8d242bcb15c8ee0b465e51479\": rpc error: code = NotFound desc = could not find container \"c2062ed9ec3ecb88cc205985959891486b1d3dc8d242bcb15c8ee0b465e51479\": container with ID starting with c2062ed9ec3ecb88cc205985959891486b1d3dc8d242bcb15c8ee0b465e51479 not found: ID does not exist"
Oct 30 14:11:10.391726 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:10.391684    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b42343a-61e7-468a-aae9-701026a365bc-kube-api-access-7vv6v\" (UniqueName: \"kubernetes.io/projected/6b42343a-61e7-468a-aae9-701026a365bc-kube-api-access-7vv6v\") pod \"6b42343a-61e7-468a-aae9-701026a365bc\" (UID: \"6b42343a-61e7-468a-aae9-701026a365bc\") "
Oct 30 14:11:10.391948 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:10.391817    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b42343a-61e7-468a-aae9-701026a365bc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b42343a-61e7-468a-aae9-701026a365bc-openshift-service-ca\") pod \"6b42343a-61e7-468a-aae9-701026a365bc\" (UID: \"6b42343a-61e7-468a-aae9-701026a365bc\") "
Oct 30 14:11:10.391948 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:10.391861    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b42343a-61e7-468a-aae9-701026a365bc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b42343a-61e7-468a-aae9-701026a365bc-metrics-certs\") pod \"6b42343a-61e7-468a-aae9-701026a365bc\" (UID: \"6b42343a-61e7-468a-aae9-701026a365bc\") "
Oct 30 14:11:10.391948 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:10.391937    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b42343a-61e7-468a-aae9-701026a365bc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b42343a-61e7-468a-aae9-701026a365bc-maas-api-tls\") pod \"6b42343a-61e7-468a-aae9-701026a365bc\" (UID: \"6b42343a-61e7-468a-aae9-701026a365bc\") "
Oct 30 14:11:10.392428 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:10.392400    5093 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/991b571e-8f93-47b0-9bb8-ab71ee26897c-model-cache\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:11:10.392568 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:10.392432    5093 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/991b571e-8f93-47b0-9bb8-ab71ee26897c-dshm\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:11:10.392568 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:10.392446    5093 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/991b571e-8f93-47b0-9bb8-ab71ee26897c-home\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:11:10.392568 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:10.392458    5093 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/991b571e-8f93-47b0-9bb8-ab71ee26897c-tmp-dir\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:11:10.392568 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:10.392470    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgpws\" (UniqueName: \"kubernetes.io/projected/991b571e-8f93-47b0-9bb8-ab71ee26897c-kube-api-access-vgpws\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:11:10.392568 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:10.392485    5093 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/991b571e-8f93-47b0-9bb8-ab71ee26897c-tls-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:11:10.394259 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:10.394226    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b42343a-61e7-468a-aae9-701026a365bc-openshift-service-ca" pod "6b42343a-61e7-468a-aae9-701026a365bc" (UID: "6b42343a-61e7-468a-aae9-701026a365bc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:11:10.397432 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:10.397386    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b42343a-61e7-468a-aae9-701026a365bc-maas-api-tls" pod "6b42343a-61e7-468a-aae9-701026a365bc" (UID: "6b42343a-61e7-468a-aae9-701026a365bc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:11:10.398592 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:10.398551    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b42343a-61e7-468a-aae9-701026a365bc-metrics-certs" pod "6b42343a-61e7-468a-aae9-701026a365bc" (UID: "6b42343a-61e7-468a-aae9-701026a365bc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:11:10.399568 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:10.399541    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b42343a-61e7-468a-aae9-701026a365bc-kube-api-access-7vv6v" pod "6b42343a-61e7-468a-aae9-701026a365bc" (UID: "6b42343a-61e7-468a-aae9-701026a365bc"). InnerVolumeSpecName "kube-api-access-7vv6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:11:10.413473 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:10.413439    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-f7ee8a/rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn"]
Oct 30 14:11:10.420165 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:10.420100    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-f7ee8a/rate-test-model-f7ee8a-kserve-8587b49fb5-xc4mn"]
Oct 30 14:11:10.493822 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:10.493730    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vv6v\" (UniqueName: \"kubernetes.io/projected/6b42343a-61e7-468a-aae9-701026a365bc-kube-api-access-7vv6v\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:11:10.493822 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:10.493765    5093 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b42343a-61e7-468a-aae9-701026a365bc-openshift-service-ca\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:11:10.493822 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:10.493778    5093 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b42343a-61e7-468a-aae9-701026a365bc-metrics-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:11:10.493822 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:10.493786    5093 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b42343a-61e7-468a-aae9-701026a365bc-maas-api-tls\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:11:11.002600 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:11.002562    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="991b571e-8f93-47b0-9bb8-ab71ee26897c" path="/var/lib/kubelet/pods/991b571e-8f93-47b0-9bb8-ab71ee26897c/volumes"
Oct 30 14:11:11.325012 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:11.324930    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c3efb2-6d4548b87-n444z" event={"ID":"6b42343a-61e7-468a-aae9-701026a365bc","Type":"ContainerDied","Data":"25814f9722072da28fcd46f33c879cd4931b2134b56084edb640be0fa613efee"}
Oct 30 14:11:11.325012 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:11.324991    5093 scope.go:122] "RemoveContainer" containerID="63843bd2872be329fb9394fdc18a1b39682ce01d73eea1d94f325d0870b96669"
Oct 30 14:11:11.325342 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:11.324939    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c3efb2-6d4548b87-n444z"
Oct 30 14:11:11.383028 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:11.382981    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c3efb2-6d4548b87-n444z"]
Oct 30 14:11:11.395466 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:11.395432    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c3efb2-6d4548b87-n444z"]
Oct 30 14:11:12.212254 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:12.212192    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c3efb2-5f44b7cbcf-4zkz8" podUID="21032810-cf90-4fef-816b-d66bf485aefb" containerName="payload-pre-processing" containerID="cri-o://02dc8a6107e3d6853c3f2eb19fc4a0b69977f60ff3ae46bed49e604c9df1174c" gracePeriod=25
Oct 30 14:11:12.999759 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:12.999720    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b42343a-61e7-468a-aae9-701026a365bc" path="/var/lib/kubelet/pods/6b42343a-61e7-468a-aae9-701026a365bc/volumes"
Oct 30 14:11:17.665745 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:17.665701    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c3efb2-5f44b7cbcf-4zkz8" podUID="21032810-cf90-4fef-816b-d66bf485aefb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:11:19.634596 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:19.634552    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-c684d2/sub-test-model-c684d2-kserve-575f6df8dc-z5xws"]
Oct 30 14:11:21.168052 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:11:21.168011    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 14:11:27.665357 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:27.665313    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c3efb2-5f44b7cbcf-4zkz8" podUID="21032810-cf90-4fef-816b-d66bf485aefb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:11:29.485162 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:29.485105    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a75d02-7f44cf6f8c-86ss9"]
Oct 30 14:11:32.481742 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:32.481701    5093 generic.go:356] "Generic (PLEG): container finished" podID="21032810-cf90-4fef-816b-d66bf485aefb" containerID="02dc8a6107e3d6853c3f2eb19fc4a0b69977f60ff3ae46bed49e604c9df1174c" exitCode=0
Oct 30 14:11:32.482319 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:32.481763    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c3efb2-5f44b7cbcf-4zkz8" event={"ID":"21032810-cf90-4fef-816b-d66bf485aefb","Type":"ContainerDied","Data":"02dc8a6107e3d6853c3f2eb19fc4a0b69977f60ff3ae46bed49e604c9df1174c"}
Oct 30 14:11:32.482319 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:32.481812    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c3efb2-5f44b7cbcf-4zkz8" event={"ID":"21032810-cf90-4fef-816b-d66bf485aefb","Type":"ContainerDied","Data":"1a4735a52158310688eaa398d761978527ba068c28f05ba8297c41572c8e9c6c"}
Oct 30 14:11:32.482319 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:32.481834    5093 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a4735a52158310688eaa398d761978527ba068c28f05ba8297c41572c8e9c6c"
Oct 30 14:11:32.509025 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:32.508989    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c3efb2-5f44b7cbcf-4zkz8"
Oct 30 14:11:32.638573 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:32.638515    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21032810-cf90-4fef-816b-d66bf485aefb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21032810-cf90-4fef-816b-d66bf485aefb-plugins-config-volume\") pod \"21032810-cf90-4fef-816b-d66bf485aefb\" (UID: \"21032810-cf90-4fef-816b-d66bf485aefb\") "
Oct 30 14:11:32.638819 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:32.638664    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21032810-cf90-4fef-816b-d66bf485aefb-kube-api-access-bwxzm\" (UniqueName: \"kubernetes.io/projected/21032810-cf90-4fef-816b-d66bf485aefb-kube-api-access-bwxzm\") pod \"21032810-cf90-4fef-816b-d66bf485aefb\" (UID: \"21032810-cf90-4fef-816b-d66bf485aefb\") "
Oct 30 14:11:32.639038 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:32.639014    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21032810-cf90-4fef-816b-d66bf485aefb-plugins-config-volume" pod "21032810-cf90-4fef-816b-d66bf485aefb" (UID: "21032810-cf90-4fef-816b-d66bf485aefb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:11:32.640842 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:32.640820    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21032810-cf90-4fef-816b-d66bf485aefb-kube-api-access-bwxzm" pod "21032810-cf90-4fef-816b-d66bf485aefb" (UID: "21032810-cf90-4fef-816b-d66bf485aefb"). InnerVolumeSpecName "kube-api-access-bwxzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:11:32.740718 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:32.740619    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwxzm\" (UniqueName: \"kubernetes.io/projected/21032810-cf90-4fef-816b-d66bf485aefb-kube-api-access-bwxzm\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:11:32.740718 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:32.740656    5093 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21032810-cf90-4fef-816b-d66bf485aefb-plugins-config-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:11:33.487757 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:33.487725    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c3efb2-5f44b7cbcf-4zkz8"
Oct 30 14:11:33.543016 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:33.542978    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c3efb2-5f44b7cbcf-4zkz8"]
Oct 30 14:11:33.556032 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:33.555997    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c3efb2-5f44b7cbcf-4zkz8"]
Oct 30 14:11:34.485757 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:34.485684    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a75d02-7f44cf6f8c-86ss9" podUID="fa8178a9-663a-44bd-ba75-c9c521f3045e" containerName="payload-pre-processing" containerID="cri-o://0865d25b7d4e55f280de43c5fc67437c9d7b9a54a94160e4deb9ff0d6cfc118f" gracePeriod=25
Oct 30 14:11:34.656202 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:34.656156    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-c684d2/sub-test-model-c684d2-kserve-575f6df8dc-z5xws" podUID="34ea00b9-9cd9-4ee1-8704-fbd22eadaede" containerName="main" containerID="cri-o://323afc65de4d0e2f37e669c14455096fc42b7868af00f7893a82336e96b54120" gracePeriod=45
Oct 30 14:11:34.935869 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:34.935830    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-c684d2/sub-test-model-c684d2-kserve-575f6df8dc-z5xws"
Oct 30 14:11:35.003484 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:35.003439    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21032810-cf90-4fef-816b-d66bf485aefb" path="/var/lib/kubelet/pods/21032810-cf90-4fef-816b-d66bf485aefb/volumes"
Oct 30 14:11:35.067716 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:35.067607    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34ea00b9-9cd9-4ee1-8704-fbd22eadaede-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34ea00b9-9cd9-4ee1-8704-fbd22eadaede-tmp-dir\") pod \"34ea00b9-9cd9-4ee1-8704-fbd22eadaede\" (UID: \"34ea00b9-9cd9-4ee1-8704-fbd22eadaede\") "
Oct 30 14:11:35.067716 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:35.067670    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34ea00b9-9cd9-4ee1-8704-fbd22eadaede-tls-certs\" (UniqueName: \"kubernetes.io/secret/34ea00b9-9cd9-4ee1-8704-fbd22eadaede-tls-certs\") pod \"34ea00b9-9cd9-4ee1-8704-fbd22eadaede\" (UID: \"34ea00b9-9cd9-4ee1-8704-fbd22eadaede\") "
Oct 30 14:11:35.067716 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:35.067706    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34ea00b9-9cd9-4ee1-8704-fbd22eadaede-dshm\" (UniqueName: \"kubernetes.io/empty-dir/34ea00b9-9cd9-4ee1-8704-fbd22eadaede-dshm\") pod \"34ea00b9-9cd9-4ee1-8704-fbd22eadaede\" (UID: \"34ea00b9-9cd9-4ee1-8704-fbd22eadaede\") "
Oct 30 14:11:35.068029 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:35.067742    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34ea00b9-9cd9-4ee1-8704-fbd22eadaede-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34ea00b9-9cd9-4ee1-8704-fbd22eadaede-model-cache\") pod \"34ea00b9-9cd9-4ee1-8704-fbd22eadaede\" (UID: \"34ea00b9-9cd9-4ee1-8704-fbd22eadaede\") "
Oct 30 14:11:35.068029 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:35.067849    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34ea00b9-9cd9-4ee1-8704-fbd22eadaede-home\" (UniqueName: \"kubernetes.io/empty-dir/34ea00b9-9cd9-4ee1-8704-fbd22eadaede-home\") pod \"34ea00b9-9cd9-4ee1-8704-fbd22eadaede\" (UID: \"34ea00b9-9cd9-4ee1-8704-fbd22eadaede\") "
Oct 30 14:11:35.068029 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:35.067973    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34ea00b9-9cd9-4ee1-8704-fbd22eadaede-kube-api-access-jmdbh\" (UniqueName: \"kubernetes.io/projected/34ea00b9-9cd9-4ee1-8704-fbd22eadaede-kube-api-access-jmdbh\") pod \"34ea00b9-9cd9-4ee1-8704-fbd22eadaede\" (UID: \"34ea00b9-9cd9-4ee1-8704-fbd22eadaede\") "
Oct 30 14:11:35.068246 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:35.068107    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34ea00b9-9cd9-4ee1-8704-fbd22eadaede-tmp-dir" pod "34ea00b9-9cd9-4ee1-8704-fbd22eadaede" (UID: "34ea00b9-9cd9-4ee1-8704-fbd22eadaede"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:11:35.068246 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:35.068119    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34ea00b9-9cd9-4ee1-8704-fbd22eadaede-model-cache" pod "34ea00b9-9cd9-4ee1-8704-fbd22eadaede" (UID: "34ea00b9-9cd9-4ee1-8704-fbd22eadaede"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:11:35.068369 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:35.068336    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34ea00b9-9cd9-4ee1-8704-fbd22eadaede-home" pod "34ea00b9-9cd9-4ee1-8704-fbd22eadaede" (UID: "34ea00b9-9cd9-4ee1-8704-fbd22eadaede"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:11:35.068906 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:35.068879    5093 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34ea00b9-9cd9-4ee1-8704-fbd22eadaede-tmp-dir\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:11:35.068906 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:35.068907    5093 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34ea00b9-9cd9-4ee1-8704-fbd22eadaede-model-cache\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:11:35.069165 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:35.068923    5093 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34ea00b9-9cd9-4ee1-8704-fbd22eadaede-home\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:11:35.070422 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:35.070399    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34ea00b9-9cd9-4ee1-8704-fbd22eadaede-dshm" pod "34ea00b9-9cd9-4ee1-8704-fbd22eadaede" (UID: "34ea00b9-9cd9-4ee1-8704-fbd22eadaede"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:11:35.070683 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:35.070663    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34ea00b9-9cd9-4ee1-8704-fbd22eadaede-tls-certs" pod "34ea00b9-9cd9-4ee1-8704-fbd22eadaede" (UID: "34ea00b9-9cd9-4ee1-8704-fbd22eadaede"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:11:35.070981 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:35.070959    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34ea00b9-9cd9-4ee1-8704-fbd22eadaede-kube-api-access-jmdbh" pod "34ea00b9-9cd9-4ee1-8704-fbd22eadaede" (UID: "34ea00b9-9cd9-4ee1-8704-fbd22eadaede"). InnerVolumeSpecName "kube-api-access-jmdbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:11:35.169932 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:35.169887    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmdbh\" (UniqueName: \"kubernetes.io/projected/34ea00b9-9cd9-4ee1-8704-fbd22eadaede-kube-api-access-jmdbh\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:11:35.169932 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:35.169932    5093 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34ea00b9-9cd9-4ee1-8704-fbd22eadaede-tls-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:11:35.170239 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:35.169950    5093 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34ea00b9-9cd9-4ee1-8704-fbd22eadaede-dshm\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:11:35.500679 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:35.500642    5093 generic.go:356] "Generic (PLEG): container finished" podID="34ea00b9-9cd9-4ee1-8704-fbd22eadaede" containerID="323afc65de4d0e2f37e669c14455096fc42b7868af00f7893a82336e96b54120" exitCode=0
Oct 30 14:11:35.501008 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:35.500698    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-c684d2/sub-test-model-c684d2-kserve-575f6df8dc-z5xws" event={"ID":"34ea00b9-9cd9-4ee1-8704-fbd22eadaede","Type":"ContainerDied","Data":"323afc65de4d0e2f37e669c14455096fc42b7868af00f7893a82336e96b54120"}
Oct 30 14:11:35.501008 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:35.500710    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-c684d2/sub-test-model-c684d2-kserve-575f6df8dc-z5xws"
Oct 30 14:11:35.501008 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:35.500738    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-c684d2/sub-test-model-c684d2-kserve-575f6df8dc-z5xws" event={"ID":"34ea00b9-9cd9-4ee1-8704-fbd22eadaede","Type":"ContainerDied","Data":"f5063c16a823175a2030f6e2d51c073ad3694c8d674dce00a068ee8491d02810"}
Oct 30 14:11:35.501008 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:35.500757    5093 scope.go:122] "RemoveContainer" containerID="323afc65de4d0e2f37e669c14455096fc42b7868af00f7893a82336e96b54120"
Oct 30 14:11:35.511687 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:35.511660    5093 scope.go:122] "RemoveContainer" containerID="323afc65de4d0e2f37e669c14455096fc42b7868af00f7893a82336e96b54120"
Oct 30 14:11:35.512017 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:11:35.511984    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"323afc65de4d0e2f37e669c14455096fc42b7868af00f7893a82336e96b54120\": container with ID starting with 323afc65de4d0e2f37e669c14455096fc42b7868af00f7893a82336e96b54120 not found: ID does not exist" containerID="323afc65de4d0e2f37e669c14455096fc42b7868af00f7893a82336e96b54120"
Oct 30 14:11:35.512083 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:35.512029    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"323afc65de4d0e2f37e669c14455096fc42b7868af00f7893a82336e96b54120"} err="failed to get container status \"323afc65de4d0e2f37e669c14455096fc42b7868af00f7893a82336e96b54120\": rpc error: code = NotFound desc = could not find container \"323afc65de4d0e2f37e669c14455096fc42b7868af00f7893a82336e96b54120\": container with ID starting with 323afc65de4d0e2f37e669c14455096fc42b7868af00f7893a82336e96b54120 not found: ID does not exist"
Oct 30 14:11:35.555267 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:35.555229    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-c684d2/sub-test-model-c684d2-kserve-575f6df8dc-z5xws"]
Oct 30 14:11:35.558320 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:35.558287    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-c684d2/sub-test-model-c684d2-kserve-575f6df8dc-z5xws"]
Oct 30 14:11:36.200351 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:36.200309    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a75d02-79d466cff8-64rb8"]
Oct 30 14:11:36.200909 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:36.200557    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a75d02-79d466cff8-64rb8" podUID="dba480c3-b59a-4acb-a634-6aa1f50de647" containerName="maas-api" containerID="cri-o://04dbb041b73d8d2878a21197c8ccae16f0c2bb3cb1a24355b453b1e7ed8b84f4" gracePeriod=30
Oct 30 14:11:36.490807 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:36.490773    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a75d02-79d466cff8-64rb8"
Oct 30 14:11:36.508063 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:36.508025    5093 generic.go:356] "Generic (PLEG): container finished" podID="dba480c3-b59a-4acb-a634-6aa1f50de647" containerID="04dbb041b73d8d2878a21197c8ccae16f0c2bb3cb1a24355b453b1e7ed8b84f4" exitCode=0
Oct 30 14:11:36.508277 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:36.508077    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a75d02-79d466cff8-64rb8"
Oct 30 14:11:36.508277 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:36.508112    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a75d02-79d466cff8-64rb8" event={"ID":"dba480c3-b59a-4acb-a634-6aa1f50de647","Type":"ContainerDied","Data":"04dbb041b73d8d2878a21197c8ccae16f0c2bb3cb1a24355b453b1e7ed8b84f4"}
Oct 30 14:11:36.508277 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:36.508193    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a75d02-79d466cff8-64rb8" event={"ID":"dba480c3-b59a-4acb-a634-6aa1f50de647","Type":"ContainerDied","Data":"5cd59ff5e034562f2dcd08cea13e5e346fd739b77dd7d0ef5061d0bd05990799"}
Oct 30 14:11:36.508277 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:36.508225    5093 scope.go:122] "RemoveContainer" containerID="04dbb041b73d8d2878a21197c8ccae16f0c2bb3cb1a24355b453b1e7ed8b84f4"
Oct 30 14:11:36.519291 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:36.519265    5093 scope.go:122] "RemoveContainer" containerID="04dbb041b73d8d2878a21197c8ccae16f0c2bb3cb1a24355b453b1e7ed8b84f4"
Oct 30 14:11:36.519584 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:11:36.519562    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04dbb041b73d8d2878a21197c8ccae16f0c2bb3cb1a24355b453b1e7ed8b84f4\": container with ID starting with 04dbb041b73d8d2878a21197c8ccae16f0c2bb3cb1a24355b453b1e7ed8b84f4 not found: ID does not exist" containerID="04dbb041b73d8d2878a21197c8ccae16f0c2bb3cb1a24355b453b1e7ed8b84f4"
Oct 30 14:11:36.519651 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:36.519598    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04dbb041b73d8d2878a21197c8ccae16f0c2bb3cb1a24355b453b1e7ed8b84f4"} err="failed to get container status \"04dbb041b73d8d2878a21197c8ccae16f0c2bb3cb1a24355b453b1e7ed8b84f4\": rpc error: code = NotFound desc = could not find container \"04dbb041b73d8d2878a21197c8ccae16f0c2bb3cb1a24355b453b1e7ed8b84f4\": container with ID starting with 04dbb041b73d8d2878a21197c8ccae16f0c2bb3cb1a24355b453b1e7ed8b84f4 not found: ID does not exist"
Oct 30 14:11:36.587866 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:36.587833    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dba480c3-b59a-4acb-a634-6aa1f50de647-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dba480c3-b59a-4acb-a634-6aa1f50de647-openshift-service-ca\") pod \"dba480c3-b59a-4acb-a634-6aa1f50de647\" (UID: \"dba480c3-b59a-4acb-a634-6aa1f50de647\") "
Oct 30 14:11:36.588057 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:36.587959    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dba480c3-b59a-4acb-a634-6aa1f50de647-kube-api-access-tmdmj\" (UniqueName: \"kubernetes.io/projected/dba480c3-b59a-4acb-a634-6aa1f50de647-kube-api-access-tmdmj\") pod \"dba480c3-b59a-4acb-a634-6aa1f50de647\" (UID: \"dba480c3-b59a-4acb-a634-6aa1f50de647\") "
Oct 30 14:11:36.588057 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:36.587983    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dba480c3-b59a-4acb-a634-6aa1f50de647-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dba480c3-b59a-4acb-a634-6aa1f50de647-metrics-certs\") pod \"dba480c3-b59a-4acb-a634-6aa1f50de647\" (UID: \"dba480c3-b59a-4acb-a634-6aa1f50de647\") "
Oct 30 14:11:36.588057 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:36.588000    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dba480c3-b59a-4acb-a634-6aa1f50de647-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dba480c3-b59a-4acb-a634-6aa1f50de647-maas-api-tls\") pod \"dba480c3-b59a-4acb-a634-6aa1f50de647\" (UID: \"dba480c3-b59a-4acb-a634-6aa1f50de647\") "
Oct 30 14:11:36.588312 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:36.588283    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dba480c3-b59a-4acb-a634-6aa1f50de647-openshift-service-ca" pod "dba480c3-b59a-4acb-a634-6aa1f50de647" (UID: "dba480c3-b59a-4acb-a634-6aa1f50de647"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:11:36.588572 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:36.588548    5093 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dba480c3-b59a-4acb-a634-6aa1f50de647-openshift-service-ca\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:11:36.590203 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:36.590180    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dba480c3-b59a-4acb-a634-6aa1f50de647-metrics-certs" pod "dba480c3-b59a-4acb-a634-6aa1f50de647" (UID: "dba480c3-b59a-4acb-a634-6aa1f50de647"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:11:36.590663 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:36.590632    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dba480c3-b59a-4acb-a634-6aa1f50de647-maas-api-tls" pod "dba480c3-b59a-4acb-a634-6aa1f50de647" (UID: "dba480c3-b59a-4acb-a634-6aa1f50de647"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:11:36.590761 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:36.590638    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dba480c3-b59a-4acb-a634-6aa1f50de647-kube-api-access-tmdmj" pod "dba480c3-b59a-4acb-a634-6aa1f50de647" (UID: "dba480c3-b59a-4acb-a634-6aa1f50de647"). InnerVolumeSpecName "kube-api-access-tmdmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:11:36.690110 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:36.690071    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmdmj\" (UniqueName: \"kubernetes.io/projected/dba480c3-b59a-4acb-a634-6aa1f50de647-kube-api-access-tmdmj\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:11:36.690110 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:36.690106    5093 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dba480c3-b59a-4acb-a634-6aa1f50de647-metrics-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:11:36.690110 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:36.690117    5093 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dba480c3-b59a-4acb-a634-6aa1f50de647-maas-api-tls\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:11:36.862024 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:36.861982    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a75d02-79d466cff8-64rb8"]
Oct 30 14:11:36.867607 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:36.867567    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a75d02-79d466cff8-64rb8"]
Oct 30 14:11:36.999262 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:36.999221    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34ea00b9-9cd9-4ee1-8704-fbd22eadaede" path="/var/lib/kubelet/pods/34ea00b9-9cd9-4ee1-8704-fbd22eadaede/volumes"
Oct 30 14:11:36.999751 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:36.999731    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dba480c3-b59a-4acb-a634-6aa1f50de647" path="/var/lib/kubelet/pods/dba480c3-b59a-4acb-a634-6aa1f50de647/volumes"
Oct 30 14:11:38.533179 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:38.533118    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a75d02-7f44cf6f8c-86ss9" podUID="fa8178a9-663a-44bd-ba75-c9c521f3045e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:11:48.532961 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:48.532918    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a75d02-7f44cf6f8c-86ss9" podUID="fa8178a9-663a-44bd-ba75-c9c521f3045e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:11:51.168029 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:11:51.167983    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 14:11:54.643076 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:54.643040    5093 generic.go:356] "Generic (PLEG): container finished" podID="fa8178a9-663a-44bd-ba75-c9c521f3045e" containerID="0865d25b7d4e55f280de43c5fc67437c9d7b9a54a94160e4deb9ff0d6cfc118f" exitCode=0
Oct 30 14:11:54.643521 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:54.643113    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a75d02-7f44cf6f8c-86ss9" event={"ID":"fa8178a9-663a-44bd-ba75-c9c521f3045e","Type":"ContainerDied","Data":"0865d25b7d4e55f280de43c5fc67437c9d7b9a54a94160e4deb9ff0d6cfc118f"}
Oct 30 14:11:54.774633 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:54.774592    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a75d02-7f44cf6f8c-86ss9"
Oct 30 14:11:54.789609 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:54.789535    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa8178a9-663a-44bd-ba75-c9c521f3045e-kube-api-access-wj8bz\" (UniqueName: \"kubernetes.io/projected/fa8178a9-663a-44bd-ba75-c9c521f3045e-kube-api-access-wj8bz\") pod \"fa8178a9-663a-44bd-ba75-c9c521f3045e\" (UID: \"fa8178a9-663a-44bd-ba75-c9c521f3045e\") "
Oct 30 14:11:54.789833 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:54.789702    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa8178a9-663a-44bd-ba75-c9c521f3045e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa8178a9-663a-44bd-ba75-c9c521f3045e-plugins-config-volume\") pod \"fa8178a9-663a-44bd-ba75-c9c521f3045e\" (UID: \"fa8178a9-663a-44bd-ba75-c9c521f3045e\") "
Oct 30 14:11:54.790183 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:54.790119    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa8178a9-663a-44bd-ba75-c9c521f3045e-plugins-config-volume" pod "fa8178a9-663a-44bd-ba75-c9c521f3045e" (UID: "fa8178a9-663a-44bd-ba75-c9c521f3045e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:11:54.792488 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:54.792461    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa8178a9-663a-44bd-ba75-c9c521f3045e-kube-api-access-wj8bz" pod "fa8178a9-663a-44bd-ba75-c9c521f3045e" (UID: "fa8178a9-663a-44bd-ba75-c9c521f3045e"). InnerVolumeSpecName "kube-api-access-wj8bz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:11:54.890753 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:54.890713    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj8bz\" (UniqueName: \"kubernetes.io/projected/fa8178a9-663a-44bd-ba75-c9c521f3045e-kube-api-access-wj8bz\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:11:54.890753 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:54.890747    5093 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa8178a9-663a-44bd-ba75-c9c521f3045e-plugins-config-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:11:55.652264 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:55.652230    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a75d02-7f44cf6f8c-86ss9"
Oct 30 14:11:55.652807 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:55.652261    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a75d02-7f44cf6f8c-86ss9" event={"ID":"fa8178a9-663a-44bd-ba75-c9c521f3045e","Type":"ContainerDied","Data":"1ea57ebec85c683a089ac035f6fa11c584b4619801d52f9cef7bb3f19eb3c79b"}
Oct 30 14:11:55.652807 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:55.652328    5093 scope.go:122] "RemoveContainer" containerID="0865d25b7d4e55f280de43c5fc67437c9d7b9a54a94160e4deb9ff0d6cfc118f"
Oct 30 14:11:55.709465 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:55.709431    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a75d02-7f44cf6f8c-86ss9"]
Oct 30 14:11:55.725601 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:55.725535    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a75d02-7f44cf6f8c-86ss9"]
Oct 30 14:11:56.999722 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:56.999673    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa8178a9-663a-44bd-ba75-c9c521f3045e" path="/var/lib/kubelet/pods/fa8178a9-663a-44bd-ba75-c9c521f3045e/volumes"
Oct 30 14:11:57.076194 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:11:57.076115    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-f7ee8a-586db84f58-bfjng"]
Oct 30 14:12:02.077244 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:02.077156    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f7ee8a-586db84f58-bfjng" podUID="f88008e4-5f1c-4814-94aa-619e90c5fd0c" containerName="payload-processing" containerID="cri-o://0e328bf699d341231dbbe138f3bbb701a6468a7c49f73a98031121d4041a587c" gracePeriod=25
Oct 30 14:12:03.275491 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:03.275450    5093 scope.go:122] "RemoveContainer" containerID="02dc8a6107e3d6853c3f2eb19fc4a0b69977f60ff3ae46bed49e604c9df1174c"
Oct 30 14:12:05.280367 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:05.280321    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f7ee8a-586db84f58-bfjng" podUID="f88008e4-5f1c-4814-94aa-619e90c5fd0c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:12:08.714301 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:08.714267    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp"]
Oct 30 14:12:08.714897 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:08.714495    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp" podUID="8285d6ec-8ad7-4a33-a696-d32242da2f80" containerName="istio-proxy" containerID="cri-o://cdce5bd18be49daaa841dafa956db4563e4eecd8bdb0332ad23cf63621d91484" gracePeriod=30
Oct 30 14:12:10.707789 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:10.707754    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f7ee8a-9d8fdd47c-k7slx"]
Oct 30 14:12:10.708340 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:10.708009    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f7ee8a-9d8fdd47c-k7slx" podUID="f2075f70-e05f-49de-92f2-aa2bbc2e28e7" containerName="maas-api" containerID="cri-o://cffe0abaa9ef85e4872ca542f099ee95b0f2d2e19e75b8b0fb8167d2ef1470da" gracePeriod=30
Oct 30 14:12:10.998839 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:10.998809    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f7ee8a-9d8fdd47c-k7slx"
Oct 30 14:12:11.067485 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:11.067435    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2075f70-e05f-49de-92f2-aa2bbc2e28e7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2075f70-e05f-49de-92f2-aa2bbc2e28e7-openshift-service-ca\") pod \"f2075f70-e05f-49de-92f2-aa2bbc2e28e7\" (UID: \"f2075f70-e05f-49de-92f2-aa2bbc2e28e7\") "
Oct 30 14:12:11.067696 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:11.067542    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2075f70-e05f-49de-92f2-aa2bbc2e28e7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2075f70-e05f-49de-92f2-aa2bbc2e28e7-metrics-certs\") pod \"f2075f70-e05f-49de-92f2-aa2bbc2e28e7\" (UID: \"f2075f70-e05f-49de-92f2-aa2bbc2e28e7\") "
Oct 30 14:12:11.067696 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:11.067655    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2075f70-e05f-49de-92f2-aa2bbc2e28e7-kube-api-access-4wv9w\" (UniqueName: \"kubernetes.io/projected/f2075f70-e05f-49de-92f2-aa2bbc2e28e7-kube-api-access-4wv9w\") pod \"f2075f70-e05f-49de-92f2-aa2bbc2e28e7\" (UID: \"f2075f70-e05f-49de-92f2-aa2bbc2e28e7\") "
Oct 30 14:12:11.067823 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:11.067756    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2075f70-e05f-49de-92f2-aa2bbc2e28e7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2075f70-e05f-49de-92f2-aa2bbc2e28e7-maas-api-tls\") pod \"f2075f70-e05f-49de-92f2-aa2bbc2e28e7\" (UID: \"f2075f70-e05f-49de-92f2-aa2bbc2e28e7\") "
Oct 30 14:12:11.067909 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:11.067867    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2075f70-e05f-49de-92f2-aa2bbc2e28e7-openshift-service-ca" pod "f2075f70-e05f-49de-92f2-aa2bbc2e28e7" (UID: "f2075f70-e05f-49de-92f2-aa2bbc2e28e7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:12:11.069207 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:11.069082    5093 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2075f70-e05f-49de-92f2-aa2bbc2e28e7-openshift-service-ca\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:12:11.070398 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:11.070366    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2075f70-e05f-49de-92f2-aa2bbc2e28e7-metrics-certs" pod "f2075f70-e05f-49de-92f2-aa2bbc2e28e7" (UID: "f2075f70-e05f-49de-92f2-aa2bbc2e28e7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:12:11.070634 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:11.070610    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2075f70-e05f-49de-92f2-aa2bbc2e28e7-kube-api-access-4wv9w" pod "f2075f70-e05f-49de-92f2-aa2bbc2e28e7" (UID: "f2075f70-e05f-49de-92f2-aa2bbc2e28e7"). InnerVolumeSpecName "kube-api-access-4wv9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:12:11.071252 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:11.071229    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2075f70-e05f-49de-92f2-aa2bbc2e28e7-maas-api-tls" pod "f2075f70-e05f-49de-92f2-aa2bbc2e28e7" (UID: "f2075f70-e05f-49de-92f2-aa2bbc2e28e7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:12:11.170473 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:11.170428    5093 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2075f70-e05f-49de-92f2-aa2bbc2e28e7-metrics-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:12:11.170473 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:11.170477    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wv9w\" (UniqueName: \"kubernetes.io/projected/f2075f70-e05f-49de-92f2-aa2bbc2e28e7-kube-api-access-4wv9w\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:12:11.170742 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:11.170493    5093 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2075f70-e05f-49de-92f2-aa2bbc2e28e7-maas-api-tls\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:12:11.767501 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:11.767458    5093 generic.go:356] "Generic (PLEG): container finished" podID="f2075f70-e05f-49de-92f2-aa2bbc2e28e7" containerID="cffe0abaa9ef85e4872ca542f099ee95b0f2d2e19e75b8b0fb8167d2ef1470da" exitCode=0
Oct 30 14:12:11.767982 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:11.767508    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f7ee8a-9d8fdd47c-k7slx"
Oct 30 14:12:11.767982 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:11.767539    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f7ee8a-9d8fdd47c-k7slx" event={"ID":"f2075f70-e05f-49de-92f2-aa2bbc2e28e7","Type":"ContainerDied","Data":"cffe0abaa9ef85e4872ca542f099ee95b0f2d2e19e75b8b0fb8167d2ef1470da"}
Oct 30 14:12:11.767982 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:11.767610    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f7ee8a-9d8fdd47c-k7slx" event={"ID":"f2075f70-e05f-49de-92f2-aa2bbc2e28e7","Type":"ContainerDied","Data":"9ed55b57795847390a10146d14f6a0efd0457d336febf04b6a2891198222dde4"}
Oct 30 14:12:11.767982 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:11.767642    5093 scope.go:122] "RemoveContainer" containerID="cffe0abaa9ef85e4872ca542f099ee95b0f2d2e19e75b8b0fb8167d2ef1470da"
Oct 30 14:12:11.778810 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:11.778788    5093 scope.go:122] "RemoveContainer" containerID="cffe0abaa9ef85e4872ca542f099ee95b0f2d2e19e75b8b0fb8167d2ef1470da"
Oct 30 14:12:11.779112 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:12:11.779086    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cffe0abaa9ef85e4872ca542f099ee95b0f2d2e19e75b8b0fb8167d2ef1470da\": container with ID starting with cffe0abaa9ef85e4872ca542f099ee95b0f2d2e19e75b8b0fb8167d2ef1470da not found: ID does not exist" containerID="cffe0abaa9ef85e4872ca542f099ee95b0f2d2e19e75b8b0fb8167d2ef1470da"
Oct 30 14:12:11.779275 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:11.779125    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cffe0abaa9ef85e4872ca542f099ee95b0f2d2e19e75b8b0fb8167d2ef1470da"} err="failed to get container status \"cffe0abaa9ef85e4872ca542f099ee95b0f2d2e19e75b8b0fb8167d2ef1470da\": rpc error: code = NotFound desc = could not find container \"cffe0abaa9ef85e4872ca542f099ee95b0f2d2e19e75b8b0fb8167d2ef1470da\": container with ID starting with cffe0abaa9ef85e4872ca542f099ee95b0f2d2e19e75b8b0fb8167d2ef1470da not found: ID does not exist"
Oct 30 14:12:11.821034 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:11.820998    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f7ee8a-9d8fdd47c-k7slx"]
Oct 30 14:12:11.835270 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:11.835236    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f7ee8a-9d8fdd47c-k7slx"]
Oct 30 14:12:12.998644 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:12.998608    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2075f70-e05f-49de-92f2-aa2bbc2e28e7" path="/var/lib/kubelet/pods/f2075f70-e05f-49de-92f2-aa2bbc2e28e7/volumes"
Oct 30 14:12:13.972990 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:13.972955    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp"
Oct 30 14:12:14.102945 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:14.102855    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8285d6ec-8ad7-4a33-a696-d32242da2f80-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8285d6ec-8ad7-4a33-a696-d32242da2f80-credential-socket\") pod \"8285d6ec-8ad7-4a33-a696-d32242da2f80\" (UID: \"8285d6ec-8ad7-4a33-a696-d32242da2f80\") "
Oct 30 14:12:14.102945 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:14.102919    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8285d6ec-8ad7-4a33-a696-d32242da2f80-istio-token\" (UniqueName: \"kubernetes.io/projected/8285d6ec-8ad7-4a33-a696-d32242da2f80-istio-token\") pod \"8285d6ec-8ad7-4a33-a696-d32242da2f80\" (UID: \"8285d6ec-8ad7-4a33-a696-d32242da2f80\") "
Oct 30 14:12:14.103544 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:14.102983    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8285d6ec-8ad7-4a33-a696-d32242da2f80-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8285d6ec-8ad7-4a33-a696-d32242da2f80-istio-envoy\") pod \"8285d6ec-8ad7-4a33-a696-d32242da2f80\" (UID: \"8285d6ec-8ad7-4a33-a696-d32242da2f80\") "
Oct 30 14:12:14.103544 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:14.103013    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8285d6ec-8ad7-4a33-a696-d32242da2f80-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8285d6ec-8ad7-4a33-a696-d32242da2f80-istiod-ca-cert\") pod \"8285d6ec-8ad7-4a33-a696-d32242da2f80\" (UID: \"8285d6ec-8ad7-4a33-a696-d32242da2f80\") "
Oct 30 14:12:14.103544 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:14.103088    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8285d6ec-8ad7-4a33-a696-d32242da2f80-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8285d6ec-8ad7-4a33-a696-d32242da2f80-istio-data\") pod \"8285d6ec-8ad7-4a33-a696-d32242da2f80\" (UID: \"8285d6ec-8ad7-4a33-a696-d32242da2f80\") "
Oct 30 14:12:14.103544 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:14.103154    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8285d6ec-8ad7-4a33-a696-d32242da2f80-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8285d6ec-8ad7-4a33-a696-d32242da2f80-workload-certs\") pod \"8285d6ec-8ad7-4a33-a696-d32242da2f80\" (UID: \"8285d6ec-8ad7-4a33-a696-d32242da2f80\") "
Oct 30 14:12:14.103544 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:14.103210    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8285d6ec-8ad7-4a33-a696-d32242da2f80-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8285d6ec-8ad7-4a33-a696-d32242da2f80-istio-podinfo\") pod \"8285d6ec-8ad7-4a33-a696-d32242da2f80\" (UID: \"8285d6ec-8ad7-4a33-a696-d32242da2f80\") "
Oct 30 14:12:14.103544 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:14.103245    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8285d6ec-8ad7-4a33-a696-d32242da2f80-kube-api-access-2c8pm\" (UniqueName: \"kubernetes.io/projected/8285d6ec-8ad7-4a33-a696-d32242da2f80-kube-api-access-2c8pm\") pod \"8285d6ec-8ad7-4a33-a696-d32242da2f80\" (UID: \"8285d6ec-8ad7-4a33-a696-d32242da2f80\") "
Oct 30 14:12:14.103544 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:14.103256    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8285d6ec-8ad7-4a33-a696-d32242da2f80-credential-socket" pod "8285d6ec-8ad7-4a33-a696-d32242da2f80" (UID: "8285d6ec-8ad7-4a33-a696-d32242da2f80"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:14.103544 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:14.103318    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8285d6ec-8ad7-4a33-a696-d32242da2f80-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8285d6ec-8ad7-4a33-a696-d32242da2f80-workload-socket\") pod \"8285d6ec-8ad7-4a33-a696-d32242da2f80\" (UID: \"8285d6ec-8ad7-4a33-a696-d32242da2f80\") "
Oct 30 14:12:14.103544 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:14.103485    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8285d6ec-8ad7-4a33-a696-d32242da2f80-workload-certs" pod "8285d6ec-8ad7-4a33-a696-d32242da2f80" (UID: "8285d6ec-8ad7-4a33-a696-d32242da2f80"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:14.103544 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:14.103508    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8285d6ec-8ad7-4a33-a696-d32242da2f80-istiod-ca-cert" pod "8285d6ec-8ad7-4a33-a696-d32242da2f80" (UID: "8285d6ec-8ad7-4a33-a696-d32242da2f80"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:12:14.104092 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:14.103867    5093 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8285d6ec-8ad7-4a33-a696-d32242da2f80-istiod-ca-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:12:14.104092 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:14.103887    5093 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8285d6ec-8ad7-4a33-a696-d32242da2f80-workload-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:12:14.104092 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:14.103901    5093 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8285d6ec-8ad7-4a33-a696-d32242da2f80-credential-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:12:14.104285 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:14.104211    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8285d6ec-8ad7-4a33-a696-d32242da2f80-istio-data" pod "8285d6ec-8ad7-4a33-a696-d32242da2f80" (UID: "8285d6ec-8ad7-4a33-a696-d32242da2f80"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:14.104285 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:14.104232    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8285d6ec-8ad7-4a33-a696-d32242da2f80-workload-socket" pod "8285d6ec-8ad7-4a33-a696-d32242da2f80" (UID: "8285d6ec-8ad7-4a33-a696-d32242da2f80"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:14.106064 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:14.106028    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8285d6ec-8ad7-4a33-a696-d32242da2f80-istio-envoy" pod "8285d6ec-8ad7-4a33-a696-d32242da2f80" (UID: "8285d6ec-8ad7-4a33-a696-d32242da2f80"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:14.106233 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:14.106091    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8285d6ec-8ad7-4a33-a696-d32242da2f80-istio-podinfo" pod "8285d6ec-8ad7-4a33-a696-d32242da2f80" (UID: "8285d6ec-8ad7-4a33-a696-d32242da2f80"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:12:14.106233 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:14.106169    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8285d6ec-8ad7-4a33-a696-d32242da2f80-istio-token" pod "8285d6ec-8ad7-4a33-a696-d32242da2f80" (UID: "8285d6ec-8ad7-4a33-a696-d32242da2f80"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:12:14.106331 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:14.106281    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8285d6ec-8ad7-4a33-a696-d32242da2f80-kube-api-access-2c8pm" pod "8285d6ec-8ad7-4a33-a696-d32242da2f80" (UID: "8285d6ec-8ad7-4a33-a696-d32242da2f80"). InnerVolumeSpecName "kube-api-access-2c8pm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:12:14.204829 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:14.204785    5093 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8285d6ec-8ad7-4a33-a696-d32242da2f80-istio-data\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:12:14.204829 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:14.204823    5093 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8285d6ec-8ad7-4a33-a696-d32242da2f80-istio-podinfo\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:12:14.204829 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:14.204832    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c8pm\" (UniqueName: \"kubernetes.io/projected/8285d6ec-8ad7-4a33-a696-d32242da2f80-kube-api-access-2c8pm\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:12:14.205080 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:14.204842    5093 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8285d6ec-8ad7-4a33-a696-d32242da2f80-workload-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:12:14.205080 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:14.204851    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8285d6ec-8ad7-4a33-a696-d32242da2f80-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:12:14.205080 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:14.204859    5093 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8285d6ec-8ad7-4a33-a696-d32242da2f80-istio-envoy\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:12:14.790870 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:14.790829    5093 generic.go:356] "Generic (PLEG): container finished" podID="8285d6ec-8ad7-4a33-a696-d32242da2f80" containerID="cdce5bd18be49daaa841dafa956db4563e4eecd8bdb0332ad23cf63621d91484" exitCode=0
Oct 30 14:12:14.790870 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:14.790879    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp"
Oct 30 14:12:14.791268 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:14.790912    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp" event={"ID":"8285d6ec-8ad7-4a33-a696-d32242da2f80","Type":"ContainerDied","Data":"cdce5bd18be49daaa841dafa956db4563e4eecd8bdb0332ad23cf63621d91484"}
Oct 30 14:12:14.791268 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:14.790985    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp" event={"ID":"8285d6ec-8ad7-4a33-a696-d32242da2f80","Type":"ContainerDied","Data":"018d07c6acc575d4a0da1e956ce6dc7905628931f121fba03a9dbd8ea727a83c"}
Oct 30 14:12:14.791268 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:14.791015    5093 scope.go:122] "RemoveContainer" containerID="cdce5bd18be49daaa841dafa956db4563e4eecd8bdb0332ad23cf63621d91484"
Oct 30 14:12:14.802406 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:14.802384    5093 scope.go:122] "RemoveContainer" containerID="58cb4ccd0578760096ff3f24e0e118b4a3cbe5cdc22db71b7f3feb99caa1207b"
Oct 30 14:12:14.812011 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:14.811990    5093 scope.go:122] "RemoveContainer" containerID="cdce5bd18be49daaa841dafa956db4563e4eecd8bdb0332ad23cf63621d91484"
Oct 30 14:12:14.812333 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:12:14.812311    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdce5bd18be49daaa841dafa956db4563e4eecd8bdb0332ad23cf63621d91484\": container with ID starting with cdce5bd18be49daaa841dafa956db4563e4eecd8bdb0332ad23cf63621d91484 not found: ID does not exist" containerID="cdce5bd18be49daaa841dafa956db4563e4eecd8bdb0332ad23cf63621d91484"
Oct 30 14:12:14.812380 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:14.812344    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdce5bd18be49daaa841dafa956db4563e4eecd8bdb0332ad23cf63621d91484"} err="failed to get container status \"cdce5bd18be49daaa841dafa956db4563e4eecd8bdb0332ad23cf63621d91484\": rpc error: code = NotFound desc = could not find container \"cdce5bd18be49daaa841dafa956db4563e4eecd8bdb0332ad23cf63621d91484\": container with ID starting with cdce5bd18be49daaa841dafa956db4563e4eecd8bdb0332ad23cf63621d91484 not found: ID does not exist"
Oct 30 14:12:14.812380 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:14.812364    5093 scope.go:122] "RemoveContainer" containerID="58cb4ccd0578760096ff3f24e0e118b4a3cbe5cdc22db71b7f3feb99caa1207b"
Oct 30 14:12:14.812620 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:12:14.812601    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58cb4ccd0578760096ff3f24e0e118b4a3cbe5cdc22db71b7f3feb99caa1207b\": container with ID starting with 58cb4ccd0578760096ff3f24e0e118b4a3cbe5cdc22db71b7f3feb99caa1207b not found: ID does not exist" containerID="58cb4ccd0578760096ff3f24e0e118b4a3cbe5cdc22db71b7f3feb99caa1207b"
Oct 30 14:12:14.812658 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:14.812627    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58cb4ccd0578760096ff3f24e0e118b4a3cbe5cdc22db71b7f3feb99caa1207b"} err="failed to get container status \"58cb4ccd0578760096ff3f24e0e118b4a3cbe5cdc22db71b7f3feb99caa1207b\": rpc error: code = NotFound desc = could not find container \"58cb4ccd0578760096ff3f24e0e118b4a3cbe5cdc22db71b7f3feb99caa1207b\": container with ID starting with 58cb4ccd0578760096ff3f24e0e118b4a3cbe5cdc22db71b7f3feb99caa1207b not found: ID does not exist"
Oct 30 14:12:14.857837 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:14.857800    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp"]
Oct 30 14:12:14.871264 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:14.871232    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-51dc9a-openshift-default-8f6d4db4c-p5glp"]
Oct 30 14:12:14.998743 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:14.998705    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8285d6ec-8ad7-4a33-a696-d32242da2f80" path="/var/lib/kubelet/pods/8285d6ec-8ad7-4a33-a696-d32242da2f80/volumes"
Oct 30 14:12:15.281006 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:15.280964    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f7ee8a-586db84f58-bfjng" podUID="f88008e4-5f1c-4814-94aa-619e90c5fd0c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:12:21.168240 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:12:21.168197    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 14:12:22.364565 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:22.364533    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f7ee8a-586db84f58-bfjng"
Oct 30 14:12:22.495539 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:22.495494    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f88008e4-5f1c-4814-94aa-619e90c5fd0c-kube-api-access-vd682\" (UniqueName: \"kubernetes.io/projected/f88008e4-5f1c-4814-94aa-619e90c5fd0c-kube-api-access-vd682\") pod \"f88008e4-5f1c-4814-94aa-619e90c5fd0c\" (UID: \"f88008e4-5f1c-4814-94aa-619e90c5fd0c\") "
Oct 30 14:12:22.495735 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:22.495574    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f88008e4-5f1c-4814-94aa-619e90c5fd0c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f88008e4-5f1c-4814-94aa-619e90c5fd0c-plugins-config-volume\") pod \"f88008e4-5f1c-4814-94aa-619e90c5fd0c\" (UID: \"f88008e4-5f1c-4814-94aa-619e90c5fd0c\") "
Oct 30 14:12:22.495903 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:22.495885    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f88008e4-5f1c-4814-94aa-619e90c5fd0c-plugins-config-volume" pod "f88008e4-5f1c-4814-94aa-619e90c5fd0c" (UID: "f88008e4-5f1c-4814-94aa-619e90c5fd0c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:12:22.495996 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:22.495985    5093 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f88008e4-5f1c-4814-94aa-619e90c5fd0c-plugins-config-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:12:22.497788 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:22.497763    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f88008e4-5f1c-4814-94aa-619e90c5fd0c-kube-api-access-vd682" pod "f88008e4-5f1c-4814-94aa-619e90c5fd0c" (UID: "f88008e4-5f1c-4814-94aa-619e90c5fd0c"). InnerVolumeSpecName "kube-api-access-vd682". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:12:22.596905 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:22.596802    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd682\" (UniqueName: \"kubernetes.io/projected/f88008e4-5f1c-4814-94aa-619e90c5fd0c-kube-api-access-vd682\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:12:22.843016 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:22.842974    5093 generic.go:356] "Generic (PLEG): container finished" podID="f88008e4-5f1c-4814-94aa-619e90c5fd0c" containerID="0e328bf699d341231dbbe138f3bbb701a6468a7c49f73a98031121d4041a587c" exitCode=0
Oct 30 14:12:22.843238 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:22.843030    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f7ee8a-586db84f58-bfjng"
Oct 30 14:12:22.843238 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:22.843063    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f7ee8a-586db84f58-bfjng" event={"ID":"f88008e4-5f1c-4814-94aa-619e90c5fd0c","Type":"ContainerDied","Data":"0e328bf699d341231dbbe138f3bbb701a6468a7c49f73a98031121d4041a587c"}
Oct 30 14:12:22.843238 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:22.843154    5093 scope.go:122] "RemoveContainer" containerID="0e328bf699d341231dbbe138f3bbb701a6468a7c49f73a98031121d4041a587c"
Oct 30 14:12:22.843238 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:22.843158    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f7ee8a-586db84f58-bfjng" event={"ID":"f88008e4-5f1c-4814-94aa-619e90c5fd0c","Type":"ContainerDied","Data":"3c4b663410b896fcabf425405b0280c301ef7581ebec29d1d34274d6918814fb"}
Oct 30 14:12:22.855521 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:22.855498    5093 scope.go:122] "RemoveContainer" containerID="0e328bf699d341231dbbe138f3bbb701a6468a7c49f73a98031121d4041a587c"
Oct 30 14:12:22.855866 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:12:22.855843    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e328bf699d341231dbbe138f3bbb701a6468a7c49f73a98031121d4041a587c\": container with ID starting with 0e328bf699d341231dbbe138f3bbb701a6468a7c49f73a98031121d4041a587c not found: ID does not exist" containerID="0e328bf699d341231dbbe138f3bbb701a6468a7c49f73a98031121d4041a587c"
Oct 30 14:12:22.855917 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:22.855876    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e328bf699d341231dbbe138f3bbb701a6468a7c49f73a98031121d4041a587c"} err="failed to get container status \"0e328bf699d341231dbbe138f3bbb701a6468a7c49f73a98031121d4041a587c\": rpc error: code = NotFound desc = could not find container \"0e328bf699d341231dbbe138f3bbb701a6468a7c49f73a98031121d4041a587c\": container with ID starting with 0e328bf699d341231dbbe138f3bbb701a6468a7c49f73a98031121d4041a587c not found: ID does not exist"
Oct 30 14:12:22.896500 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:22.896460    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-f7ee8a-586db84f58-bfjng"]
Oct 30 14:12:22.899905 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:22.899874    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-f7ee8a-586db84f58-bfjng"]
Oct 30 14:12:22.998021 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:22.997984    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f88008e4-5f1c-4814-94aa-619e90c5fd0c" path="/var/lib/kubelet/pods/f88008e4-5f1c-4814-94aa-619e90c5fd0c/volumes"
Oct 30 14:12:29.289916 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.289879    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78ac3eda-hcrzl"]
Oct 30 14:12:29.290462 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.290435    5093 cpu_manager.go:414] "containerMap: removing container" podUID="6b42343a-61e7-468a-aae9-701026a365bc" containerName="maas-api"
Oct 30 14:12:29.290462 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.290460    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b42343a-61e7-468a-aae9-701026a365bc" containerName="maas-api"
Oct 30 14:12:29.290649 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.290472    5093 cpu_manager.go:414] "containerMap: removing container" podUID="762bb1a6-9339-4c49-bf97-1e160e7b0fc6" containerName="istio-proxy"
Oct 30 14:12:29.290649 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.290477    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="762bb1a6-9339-4c49-bf97-1e160e7b0fc6" containerName="istio-proxy"
Oct 30 14:12:29.290649 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.290497    5093 cpu_manager.go:414] "containerMap: removing container" podUID="f2075f70-e05f-49de-92f2-aa2bbc2e28e7" containerName="maas-api"
Oct 30 14:12:29.290649 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.290503    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2075f70-e05f-49de-92f2-aa2bbc2e28e7" containerName="maas-api"
Oct 30 14:12:29.290649 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.290516    5093 cpu_manager.go:414] "containerMap: removing container" podUID="991b571e-8f93-47b0-9bb8-ab71ee26897c" containerName="main"
Oct 30 14:12:29.290649 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.290523    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="991b571e-8f93-47b0-9bb8-ab71ee26897c" containerName="main"
Oct 30 14:12:29.290649 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.290541    5093 cpu_manager.go:414] "containerMap: removing container" podUID="926c412f-623c-489d-8100-c66c75a5f478" containerName="istio-proxy"
Oct 30 14:12:29.290649 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.290547    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="926c412f-623c-489d-8100-c66c75a5f478" containerName="istio-proxy"
Oct 30 14:12:29.290649 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.290555    5093 cpu_manager.go:414] "containerMap: removing container" podUID="b0d3d9d3-593a-478d-963b-d8b4b395a661" containerName="istio-proxy"
Oct 30 14:12:29.290649 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.290559    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0d3d9d3-593a-478d-963b-d8b4b395a661" containerName="istio-proxy"
Oct 30 14:12:29.290649 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.290568    5093 cpu_manager.go:414] "containerMap: removing container" podUID="f9778494-18c5-4eb6-926b-92f69cbde312" containerName="istio-proxy"
Oct 30 14:12:29.290649 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.290572    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9778494-18c5-4eb6-926b-92f69cbde312" containerName="istio-proxy"
Oct 30 14:12:29.290649 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.290580    5093 cpu_manager.go:414] "containerMap: removing container" podUID="fa8178a9-663a-44bd-ba75-c9c521f3045e" containerName="payload-pre-processing"
Oct 30 14:12:29.290649 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.290585    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa8178a9-663a-44bd-ba75-c9c521f3045e" containerName="payload-pre-processing"
Oct 30 14:12:29.290649 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.290599    5093 cpu_manager.go:414] "containerMap: removing container" podUID="f88008e4-5f1c-4814-94aa-619e90c5fd0c" containerName="payload-processing"
Oct 30 14:12:29.290649 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.290603    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f88008e4-5f1c-4814-94aa-619e90c5fd0c" containerName="payload-processing"
Oct 30 14:12:29.290649 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.290614    5093 cpu_manager.go:414] "containerMap: removing container" podUID="21032810-cf90-4fef-816b-d66bf485aefb" containerName="payload-pre-processing"
Oct 30 14:12:29.290649 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.290622    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21032810-cf90-4fef-816b-d66bf485aefb" containerName="payload-pre-processing"
Oct 30 14:12:29.290649 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.290629    5093 cpu_manager.go:414] "containerMap: removing container" podUID="8285d6ec-8ad7-4a33-a696-d32242da2f80" containerName="istio-proxy"
Oct 30 14:12:29.290649 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.290634    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8285d6ec-8ad7-4a33-a696-d32242da2f80" containerName="istio-proxy"
Oct 30 14:12:29.290649 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.290639    5093 cpu_manager.go:414] "containerMap: removing container" podUID="16483221-0a88-4aa3-a47d-6d119206b423" containerName="main"
Oct 30 14:12:29.290649 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.290643    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16483221-0a88-4aa3-a47d-6d119206b423" containerName="main"
Oct 30 14:12:29.290649 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.290652    5093 cpu_manager.go:414] "containerMap: removing container" podUID="dba480c3-b59a-4acb-a634-6aa1f50de647" containerName="maas-api"
Oct 30 14:12:29.290649 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.290656    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dba480c3-b59a-4acb-a634-6aa1f50de647" containerName="maas-api"
Oct 30 14:12:29.291853 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.290664    5093 cpu_manager.go:414] "containerMap: removing container" podUID="a51f3807-9699-4d02-8839-302a7f44afa3" containerName="test-curl-7686-4ba6d9"
Oct 30 14:12:29.291853 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.290668    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a51f3807-9699-4d02-8839-302a7f44afa3" containerName="test-curl-7686-4ba6d9"
Oct 30 14:12:29.291853 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.290676    5093 cpu_manager.go:414] "containerMap: removing container" podUID="d2c83cc2-4c02-493a-948a-3a0ecf046a37" containerName="istio-proxy"
Oct 30 14:12:29.291853 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.290680    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2c83cc2-4c02-493a-948a-3a0ecf046a37" containerName="istio-proxy"
Oct 30 14:12:29.291853 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.290706    5093 cpu_manager.go:414] "containerMap: removing container" podUID="34ea00b9-9cd9-4ee1-8704-fbd22eadaede" containerName="main"
Oct 30 14:12:29.291853 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.290711    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34ea00b9-9cd9-4ee1-8704-fbd22eadaede" containerName="main"
Oct 30 14:12:29.291853 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.290800    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="a51f3807-9699-4d02-8839-302a7f44afa3" containerName="test-curl-7686-4ba6d9"
Oct 30 14:12:29.291853 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.290808    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2c83cc2-4c02-493a-948a-3a0ecf046a37" containerName="istio-proxy"
Oct 30 14:12:29.291853 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.290815    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="34ea00b9-9cd9-4ee1-8704-fbd22eadaede" containerName="main"
Oct 30 14:12:29.291853 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.290822    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="8285d6ec-8ad7-4a33-a696-d32242da2f80" containerName="istio-proxy"
Oct 30 14:12:29.291853 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.290828    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0d3d9d3-593a-478d-963b-d8b4b395a661" containerName="istio-proxy"
Oct 30 14:12:29.291853 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.290833    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b42343a-61e7-468a-aae9-701026a365bc" containerName="maas-api"
Oct 30 14:12:29.291853 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.290839    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="762bb1a6-9339-4c49-bf97-1e160e7b0fc6" containerName="istio-proxy"
Oct 30 14:12:29.291853 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.290846    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2075f70-e05f-49de-92f2-aa2bbc2e28e7" containerName="maas-api"
Oct 30 14:12:29.291853 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.290852    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="991b571e-8f93-47b0-9bb8-ab71ee26897c" containerName="main"
Oct 30 14:12:29.291853 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.290858    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="926c412f-623c-489d-8100-c66c75a5f478" containerName="istio-proxy"
Oct 30 14:12:29.291853 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.290863    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9778494-18c5-4eb6-926b-92f69cbde312" containerName="istio-proxy"
Oct 30 14:12:29.291853 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.290872    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa8178a9-663a-44bd-ba75-c9c521f3045e" containerName="payload-pre-processing"
Oct 30 14:12:29.291853 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.290883    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="f88008e4-5f1c-4814-94aa-619e90c5fd0c" containerName="payload-processing"
Oct 30 14:12:29.291853 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.290893    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="21032810-cf90-4fef-816b-d66bf485aefb" containerName="payload-pre-processing"
Oct 30 14:12:29.291853 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.290900    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="16483221-0a88-4aa3-a47d-6d119206b423" containerName="main"
Oct 30 14:12:29.291853 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.290908    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="dba480c3-b59a-4acb-a634-6aa1f50de647" containerName="maas-api"
Oct 30 14:12:29.296075 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.296039    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78ac3eda-hcrzl"
Oct 30 14:12:29.313451 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.313411    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78ac3eda-hcrzl"]
Oct 30 14:12:29.322464 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.322429    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:12:29.322640 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.322517    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 14:12:29.365362 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.365231    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8f349f4-3e49-4c63-bc17-1d220cb7f477-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78ac3eda-hcrzl\" (UID: \"c8f349f4-3e49-4c63-bc17-1d220cb7f477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78ac3eda-hcrzl"
Oct 30 14:12:29.383542 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.365400    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8f349f4-3e49-4c63-bc17-1d220cb7f477-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78ac3eda-hcrzl\" (UID: \"c8f349f4-3e49-4c63-bc17-1d220cb7f477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78ac3eda-hcrzl"
Oct 30 14:12:29.383542 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.365471    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-552nm\" (UniqueName: \"kubernetes.io/projected/c8f349f4-3e49-4c63-bc17-1d220cb7f477-kube-api-access-552nm\") pod \"maas-api-revoke-sub-78ac3eda-hcrzl\" (UID: \"c8f349f4-3e49-4c63-bc17-1d220cb7f477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78ac3eda-hcrzl"
Oct 30 14:12:29.466970 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.466926    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8f349f4-3e49-4c63-bc17-1d220cb7f477-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78ac3eda-hcrzl\" (UID: \"c8f349f4-3e49-4c63-bc17-1d220cb7f477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78ac3eda-hcrzl"
Oct 30 14:12:29.467178 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.467028    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8f349f4-3e49-4c63-bc17-1d220cb7f477-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78ac3eda-hcrzl\" (UID: \"c8f349f4-3e49-4c63-bc17-1d220cb7f477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78ac3eda-hcrzl"
Oct 30 14:12:29.467178 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.467076    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-552nm\" (UniqueName: \"kubernetes.io/projected/c8f349f4-3e49-4c63-bc17-1d220cb7f477-kube-api-access-552nm\") pod \"maas-api-revoke-sub-78ac3eda-hcrzl\" (UID: \"c8f349f4-3e49-4c63-bc17-1d220cb7f477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78ac3eda-hcrzl"
Oct 30 14:12:29.467823 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.467797    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8f349f4-3e49-4c63-bc17-1d220cb7f477-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78ac3eda-hcrzl\" (UID: \"c8f349f4-3e49-4c63-bc17-1d220cb7f477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78ac3eda-hcrzl"
Oct 30 14:12:29.469534 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.469504    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8f349f4-3e49-4c63-bc17-1d220cb7f477-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78ac3eda-hcrzl\" (UID: \"c8f349f4-3e49-4c63-bc17-1d220cb7f477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78ac3eda-hcrzl"
Oct 30 14:12:29.489370 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.489339    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-552nm\" (UniqueName: \"kubernetes.io/projected/c8f349f4-3e49-4c63-bc17-1d220cb7f477-kube-api-access-552nm\") pod \"maas-api-revoke-sub-78ac3eda-hcrzl\" (UID: \"c8f349f4-3e49-4c63-bc17-1d220cb7f477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78ac3eda-hcrzl"
Oct 30 14:12:29.612073 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.611981    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78ac3eda-hcrzl"
Oct 30 14:12:29.934927 ip-10-0-134-31 kubenswrapper[5093]: W1030 14:12:29.934880    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8f349f4_3e49_4c63_bc17_1d220cb7f477.slice/crio-d54b227695e8e05513facb9004239b59d49e209c0e8958b5de878b6f23e488fb WatchSource:0}: Error finding container d54b227695e8e05513facb9004239b59d49e209c0e8958b5de878b6f23e488fb: Status 404 returned error can't find the container with id d54b227695e8e05513facb9004239b59d49e209c0e8958b5de878b6f23e488fb
Oct 30 14:12:29.935064 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:29.934926    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78ac3eda-hcrzl"]
Oct 30 14:12:30.905721 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:30.905687    5093 generic.go:356] "Generic (PLEG): container finished" podID="c8f349f4-3e49-4c63-bc17-1d220cb7f477" containerID="682cb988c2a45045af07c1c865fcc91bb228c420a5ea7f8a8eadf4e6cda51eb5" exitCode=0
Oct 30 14:12:30.906162 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:30.905773    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78ac3eda-hcrzl" event={"ID":"c8f349f4-3e49-4c63-bc17-1d220cb7f477","Type":"ContainerDied","Data":"682cb988c2a45045af07c1c865fcc91bb228c420a5ea7f8a8eadf4e6cda51eb5"}
Oct 30 14:12:30.906162 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:30.905825    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78ac3eda-hcrzl" event={"ID":"c8f349f4-3e49-4c63-bc17-1d220cb7f477","Type":"ContainerStarted","Data":"d54b227695e8e05513facb9004239b59d49e209c0e8958b5de878b6f23e488fb"}
Oct 30 14:12:32.075165 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:32.075120    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78ac3eda-hcrzl"
Oct 30 14:12:32.098600 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:32.098559    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8f349f4-3e49-4c63-bc17-1d220cb7f477-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8f349f4-3e49-4c63-bc17-1d220cb7f477-maas-api-openshift-ca\") pod \"c8f349f4-3e49-4c63-bc17-1d220cb7f477\" (UID: \"c8f349f4-3e49-4c63-bc17-1d220cb7f477\") "
Oct 30 14:12:32.098874 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:32.098712    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8f349f4-3e49-4c63-bc17-1d220cb7f477-kube-api-access-552nm\" (UniqueName: \"kubernetes.io/projected/c8f349f4-3e49-4c63-bc17-1d220cb7f477-kube-api-access-552nm\") pod \"c8f349f4-3e49-4c63-bc17-1d220cb7f477\" (UID: \"c8f349f4-3e49-4c63-bc17-1d220cb7f477\") "
Oct 30 14:12:32.098874 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:32.098759    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8f349f4-3e49-4c63-bc17-1d220cb7f477-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8f349f4-3e49-4c63-bc17-1d220cb7f477-maas-api-xks-ca\") pod \"c8f349f4-3e49-4c63-bc17-1d220cb7f477\" (UID: \"c8f349f4-3e49-4c63-bc17-1d220cb7f477\") "
Oct 30 14:12:32.099145 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:32.099084    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8f349f4-3e49-4c63-bc17-1d220cb7f477-maas-api-openshift-ca" pod "c8f349f4-3e49-4c63-bc17-1d220cb7f477" (UID: "c8f349f4-3e49-4c63-bc17-1d220cb7f477"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:12:32.099332 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:32.099307    5093 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8f349f4-3e49-4c63-bc17-1d220cb7f477-maas-api-openshift-ca\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:12:32.101304 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:32.101278    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8f349f4-3e49-4c63-bc17-1d220cb7f477-maas-api-xks-ca" pod "c8f349f4-3e49-4c63-bc17-1d220cb7f477" (UID: "c8f349f4-3e49-4c63-bc17-1d220cb7f477"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:12:32.101403 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:32.101303    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8f349f4-3e49-4c63-bc17-1d220cb7f477-kube-api-access-552nm" pod "c8f349f4-3e49-4c63-bc17-1d220cb7f477" (UID: "c8f349f4-3e49-4c63-bc17-1d220cb7f477"). InnerVolumeSpecName "kube-api-access-552nm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:12:32.200955 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:32.200911    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-552nm\" (UniqueName: \"kubernetes.io/projected/c8f349f4-3e49-4c63-bc17-1d220cb7f477-kube-api-access-552nm\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:12:32.200955 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:32.200948    5093 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8f349f4-3e49-4c63-bc17-1d220cb7f477-maas-api-xks-ca\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:12:32.919762 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:32.919727    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78ac3eda-hcrzl"
Oct 30 14:12:32.919964 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:32.919722    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78ac3eda-hcrzl" event={"ID":"c8f349f4-3e49-4c63-bc17-1d220cb7f477","Type":"ContainerDied","Data":"d54b227695e8e05513facb9004239b59d49e209c0e8958b5de878b6f23e488fb"}
Oct 30 14:12:32.919964 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:32.919871    5093 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d54b227695e8e05513facb9004239b59d49e209c0e8958b5de878b6f23e488fb"
Oct 30 14:12:43.424173 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:43.424068    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c684d2-568d5d66c9-xlfrb"]
Oct 30 14:12:43.424713 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:43.424348    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c684d2-568d5d66c9-xlfrb" podUID="ec7e602c-fb95-46e6-bbaa-74dd09dbebe5" containerName="maas-api" containerID="cri-o://82107c5a156eefa9712b1fdf147b43055a74a167627431e00cfe6fdb8e198939" gracePeriod=30
Oct 30 14:12:43.699249 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:43.699216    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c684d2-568d5d66c9-xlfrb"
Oct 30 14:12:43.830282 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:43.830239    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec7e602c-fb95-46e6-bbaa-74dd09dbebe5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec7e602c-fb95-46e6-bbaa-74dd09dbebe5-maas-api-tls\") pod \"ec7e602c-fb95-46e6-bbaa-74dd09dbebe5\" (UID: \"ec7e602c-fb95-46e6-bbaa-74dd09dbebe5\") "
Oct 30 14:12:43.830464 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:43.830351    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec7e602c-fb95-46e6-bbaa-74dd09dbebe5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec7e602c-fb95-46e6-bbaa-74dd09dbebe5-metrics-certs\") pod \"ec7e602c-fb95-46e6-bbaa-74dd09dbebe5\" (UID: \"ec7e602c-fb95-46e6-bbaa-74dd09dbebe5\") "
Oct 30 14:12:43.830464 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:43.830382    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec7e602c-fb95-46e6-bbaa-74dd09dbebe5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec7e602c-fb95-46e6-bbaa-74dd09dbebe5-openshift-service-ca\") pod \"ec7e602c-fb95-46e6-bbaa-74dd09dbebe5\" (UID: \"ec7e602c-fb95-46e6-bbaa-74dd09dbebe5\") "
Oct 30 14:12:43.830464 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:43.830410    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec7e602c-fb95-46e6-bbaa-74dd09dbebe5-kube-api-access-9krc5\" (UniqueName: \"kubernetes.io/projected/ec7e602c-fb95-46e6-bbaa-74dd09dbebe5-kube-api-access-9krc5\") pod \"ec7e602c-fb95-46e6-bbaa-74dd09dbebe5\" (UID: \"ec7e602c-fb95-46e6-bbaa-74dd09dbebe5\") "
Oct 30 14:12:43.830839 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:43.830801    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec7e602c-fb95-46e6-bbaa-74dd09dbebe5-openshift-service-ca" pod "ec7e602c-fb95-46e6-bbaa-74dd09dbebe5" (UID: "ec7e602c-fb95-46e6-bbaa-74dd09dbebe5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:12:43.832696 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:43.832667    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec7e602c-fb95-46e6-bbaa-74dd09dbebe5-maas-api-tls" pod "ec7e602c-fb95-46e6-bbaa-74dd09dbebe5" (UID: "ec7e602c-fb95-46e6-bbaa-74dd09dbebe5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:12:43.832810 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:43.832679    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec7e602c-fb95-46e6-bbaa-74dd09dbebe5-kube-api-access-9krc5" pod "ec7e602c-fb95-46e6-bbaa-74dd09dbebe5" (UID: "ec7e602c-fb95-46e6-bbaa-74dd09dbebe5"). InnerVolumeSpecName "kube-api-access-9krc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:12:43.832867 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:43.832845    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec7e602c-fb95-46e6-bbaa-74dd09dbebe5-metrics-certs" pod "ec7e602c-fb95-46e6-bbaa-74dd09dbebe5" (UID: "ec7e602c-fb95-46e6-bbaa-74dd09dbebe5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:12:43.932607 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:43.932482    5093 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec7e602c-fb95-46e6-bbaa-74dd09dbebe5-maas-api-tls\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:12:43.932607 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:43.932545    5093 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec7e602c-fb95-46e6-bbaa-74dd09dbebe5-metrics-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:12:43.932607 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:43.932561    5093 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec7e602c-fb95-46e6-bbaa-74dd09dbebe5-openshift-service-ca\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:12:43.932607 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:43.932578    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9krc5\" (UniqueName: \"kubernetes.io/projected/ec7e602c-fb95-46e6-bbaa-74dd09dbebe5-kube-api-access-9krc5\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:12:43.995462 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:43.995428    5093 generic.go:356] "Generic (PLEG): container finished" podID="ec7e602c-fb95-46e6-bbaa-74dd09dbebe5" containerID="82107c5a156eefa9712b1fdf147b43055a74a167627431e00cfe6fdb8e198939" exitCode=0
Oct 30 14:12:43.995677 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:43.995477    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c684d2-568d5d66c9-xlfrb" event={"ID":"ec7e602c-fb95-46e6-bbaa-74dd09dbebe5","Type":"ContainerDied","Data":"82107c5a156eefa9712b1fdf147b43055a74a167627431e00cfe6fdb8e198939"}
Oct 30 14:12:43.995677 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:43.995496    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c684d2-568d5d66c9-xlfrb"
Oct 30 14:12:43.995677 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:43.995523    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c684d2-568d5d66c9-xlfrb" event={"ID":"ec7e602c-fb95-46e6-bbaa-74dd09dbebe5","Type":"ContainerDied","Data":"e7eb7a43a2ae6390df1d1e1b6b28289656d99e3bcb9b73eca3e4275ddebaac90"}
Oct 30 14:12:43.995677 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:43.995544    5093 scope.go:122] "RemoveContainer" containerID="82107c5a156eefa9712b1fdf147b43055a74a167627431e00cfe6fdb8e198939"
Oct 30 14:12:44.007285 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:44.007257    5093 scope.go:122] "RemoveContainer" containerID="82107c5a156eefa9712b1fdf147b43055a74a167627431e00cfe6fdb8e198939"
Oct 30 14:12:44.007657 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:12:44.007636    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82107c5a156eefa9712b1fdf147b43055a74a167627431e00cfe6fdb8e198939\": container with ID starting with 82107c5a156eefa9712b1fdf147b43055a74a167627431e00cfe6fdb8e198939 not found: ID does not exist" containerID="82107c5a156eefa9712b1fdf147b43055a74a167627431e00cfe6fdb8e198939"
Oct 30 14:12:44.007732 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:44.007670    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82107c5a156eefa9712b1fdf147b43055a74a167627431e00cfe6fdb8e198939"} err="failed to get container status \"82107c5a156eefa9712b1fdf147b43055a74a167627431e00cfe6fdb8e198939\": rpc error: code = NotFound desc = could not find container \"82107c5a156eefa9712b1fdf147b43055a74a167627431e00cfe6fdb8e198939\": container with ID starting with 82107c5a156eefa9712b1fdf147b43055a74a167627431e00cfe6fdb8e198939 not found: ID does not exist"
Oct 30 14:12:44.051257 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:44.051223    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c684d2-568d5d66c9-xlfrb"]
Oct 30 14:12:44.054971 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:44.054936    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c684d2-568d5d66c9-xlfrb"]
Oct 30 14:12:44.999565 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:44.999525    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec7e602c-fb95-46e6-bbaa-74dd09dbebe5" path="/var/lib/kubelet/pods/ec7e602c-fb95-46e6-bbaa-74dd09dbebe5/volumes"
Oct 30 14:12:47.450917 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:47.450877    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6"]
Oct 30 14:12:47.451512 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:47.451111    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6" podUID="e378bf44-e323-460f-9096-fd8835ad91b8" containerName="istio-proxy" containerID="cri-o://04f1f3b38a412348f18d73b5e3ceb7bf41de1f7e199e437a04ea1aa735cd2f7b" gracePeriod=30
Oct 30 14:12:48.744097 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:48.744041    5093 patch_prober.go:29] interesting pod/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 14:12:48.744554 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:48.744112    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6" podUID="e378bf44-e323-460f-9096-fd8835ad91b8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 14:12:51.169086 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:12:51.169040    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 14:12:52.761413 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:52.761380    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6"
Oct 30 14:12:52.941524 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:52.941470    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e378bf44-e323-460f-9096-fd8835ad91b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e378bf44-e323-460f-9096-fd8835ad91b8-istio-podinfo\") pod \"e378bf44-e323-460f-9096-fd8835ad91b8\" (UID: \"e378bf44-e323-460f-9096-fd8835ad91b8\") "
Oct 30 14:12:52.941746 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:52.941545    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e378bf44-e323-460f-9096-fd8835ad91b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e378bf44-e323-460f-9096-fd8835ad91b8-workload-socket\") pod \"e378bf44-e323-460f-9096-fd8835ad91b8\" (UID: \"e378bf44-e323-460f-9096-fd8835ad91b8\") "
Oct 30 14:12:52.941746 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:52.941616    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e378bf44-e323-460f-9096-fd8835ad91b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e378bf44-e323-460f-9096-fd8835ad91b8-istiod-ca-cert\") pod \"e378bf44-e323-460f-9096-fd8835ad91b8\" (UID: \"e378bf44-e323-460f-9096-fd8835ad91b8\") "
Oct 30 14:12:52.941746 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:52.941687    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e378bf44-e323-460f-9096-fd8835ad91b8-istio-token\" (UniqueName: \"kubernetes.io/projected/e378bf44-e323-460f-9096-fd8835ad91b8-istio-token\") pod \"e378bf44-e323-460f-9096-fd8835ad91b8\" (UID: \"e378bf44-e323-460f-9096-fd8835ad91b8\") "
Oct 30 14:12:52.941746 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:52.941737    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e378bf44-e323-460f-9096-fd8835ad91b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e378bf44-e323-460f-9096-fd8835ad91b8-istio-data\") pod \"e378bf44-e323-460f-9096-fd8835ad91b8\" (UID: \"e378bf44-e323-460f-9096-fd8835ad91b8\") "
Oct 30 14:12:52.941987 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:52.941792    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e378bf44-e323-460f-9096-fd8835ad91b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e378bf44-e323-460f-9096-fd8835ad91b8-credential-socket\") pod \"e378bf44-e323-460f-9096-fd8835ad91b8\" (UID: \"e378bf44-e323-460f-9096-fd8835ad91b8\") "
Oct 30 14:12:52.941987 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:52.941827    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e378bf44-e323-460f-9096-fd8835ad91b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e378bf44-e323-460f-9096-fd8835ad91b8-workload-certs\") pod \"e378bf44-e323-460f-9096-fd8835ad91b8\" (UID: \"e378bf44-e323-460f-9096-fd8835ad91b8\") "
Oct 30 14:12:52.941987 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:52.941884    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e378bf44-e323-460f-9096-fd8835ad91b8-kube-api-access-zw2l8\" (UniqueName: \"kubernetes.io/projected/e378bf44-e323-460f-9096-fd8835ad91b8-kube-api-access-zw2l8\") pod \"e378bf44-e323-460f-9096-fd8835ad91b8\" (UID: \"e378bf44-e323-460f-9096-fd8835ad91b8\") "
Oct 30 14:12:52.941987 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:52.941906    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e378bf44-e323-460f-9096-fd8835ad91b8-workload-socket" pod "e378bf44-e323-460f-9096-fd8835ad91b8" (UID: "e378bf44-e323-460f-9096-fd8835ad91b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:52.941987 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:52.941915    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e378bf44-e323-460f-9096-fd8835ad91b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e378bf44-e323-460f-9096-fd8835ad91b8-istio-envoy\") pod \"e378bf44-e323-460f-9096-fd8835ad91b8\" (UID: \"e378bf44-e323-460f-9096-fd8835ad91b8\") "
Oct 30 14:12:52.942271 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:52.942061    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e378bf44-e323-460f-9096-fd8835ad91b8-istiod-ca-cert" pod "e378bf44-e323-460f-9096-fd8835ad91b8" (UID: "e378bf44-e323-460f-9096-fd8835ad91b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:12:52.942358 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:52.942309    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e378bf44-e323-460f-9096-fd8835ad91b8-credential-socket" pod "e378bf44-e323-460f-9096-fd8835ad91b8" (UID: "e378bf44-e323-460f-9096-fd8835ad91b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:52.942683 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:52.942437    5093 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e378bf44-e323-460f-9096-fd8835ad91b8-istiod-ca-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:12:52.942683 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:52.942459    5093 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e378bf44-e323-460f-9096-fd8835ad91b8-credential-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:12:52.942683 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:52.942473    5093 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e378bf44-e323-460f-9096-fd8835ad91b8-workload-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:12:52.942683 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:52.942624    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e378bf44-e323-460f-9096-fd8835ad91b8-istio-data" pod "e378bf44-e323-460f-9096-fd8835ad91b8" (UID: "e378bf44-e323-460f-9096-fd8835ad91b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:52.942683 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:52.942640    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e378bf44-e323-460f-9096-fd8835ad91b8-workload-certs" pod "e378bf44-e323-460f-9096-fd8835ad91b8" (UID: "e378bf44-e323-460f-9096-fd8835ad91b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:52.944503 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:52.944477    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e378bf44-e323-460f-9096-fd8835ad91b8-istio-podinfo" pod "e378bf44-e323-460f-9096-fd8835ad91b8" (UID: "e378bf44-e323-460f-9096-fd8835ad91b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:12:52.944617 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:52.944511    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e378bf44-e323-460f-9096-fd8835ad91b8-istio-token" pod "e378bf44-e323-460f-9096-fd8835ad91b8" (UID: "e378bf44-e323-460f-9096-fd8835ad91b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:12:52.944985 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:52.944964    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e378bf44-e323-460f-9096-fd8835ad91b8-istio-envoy" pod "e378bf44-e323-460f-9096-fd8835ad91b8" (UID: "e378bf44-e323-460f-9096-fd8835ad91b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:52.945063 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:52.945041    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e378bf44-e323-460f-9096-fd8835ad91b8-kube-api-access-zw2l8" pod "e378bf44-e323-460f-9096-fd8835ad91b8" (UID: "e378bf44-e323-460f-9096-fd8835ad91b8"). InnerVolumeSpecName "kube-api-access-zw2l8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:12:53.043861 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:53.043814    5093 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e378bf44-e323-460f-9096-fd8835ad91b8-istio-podinfo\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:12:53.043861 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:53.043848    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e378bf44-e323-460f-9096-fd8835ad91b8-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:12:53.043861 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:53.043861    5093 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e378bf44-e323-460f-9096-fd8835ad91b8-istio-data\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:12:53.043861 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:53.043871    5093 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e378bf44-e323-460f-9096-fd8835ad91b8-workload-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:12:53.044174 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:53.043880    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw2l8\" (UniqueName: \"kubernetes.io/projected/e378bf44-e323-460f-9096-fd8835ad91b8-kube-api-access-zw2l8\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:12:53.044174 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:53.043892    5093 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e378bf44-e323-460f-9096-fd8835ad91b8-istio-envoy\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:12:53.054260 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:53.054015    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw"]
Oct 30 14:12:53.059543 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:53.059505    5093 generic.go:356] "Generic (PLEG): container finished" podID="e378bf44-e323-460f-9096-fd8835ad91b8" containerID="04f1f3b38a412348f18d73b5e3ceb7bf41de1f7e199e437a04ea1aa735cd2f7b" exitCode=0
Oct 30 14:12:53.059736 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:53.059567    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6"
Oct 30 14:12:53.059736 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:53.059567    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6" event={"ID":"e378bf44-e323-460f-9096-fd8835ad91b8","Type":"ContainerDied","Data":"04f1f3b38a412348f18d73b5e3ceb7bf41de1f7e199e437a04ea1aa735cd2f7b"}
Oct 30 14:12:53.059736 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:53.059699    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6" event={"ID":"e378bf44-e323-460f-9096-fd8835ad91b8","Type":"ContainerDied","Data":"ea32d7e8dc3aa76870b2e51403179d26e415ca60d1fbb1fbf0eb3769c5966ad3"}
Oct 30 14:12:53.059736 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:53.059723    5093 scope.go:122] "RemoveContainer" containerID="04f1f3b38a412348f18d73b5e3ceb7bf41de1f7e199e437a04ea1aa735cd2f7b"
Oct 30 14:12:53.078218 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:53.078177    5093 scope.go:122] "RemoveContainer" containerID="04f1f3b38a412348f18d73b5e3ceb7bf41de1f7e199e437a04ea1aa735cd2f7b"
Oct 30 14:12:53.078661 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:12:53.078635    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04f1f3b38a412348f18d73b5e3ceb7bf41de1f7e199e437a04ea1aa735cd2f7b\": container with ID starting with 04f1f3b38a412348f18d73b5e3ceb7bf41de1f7e199e437a04ea1aa735cd2f7b not found: ID does not exist" containerID="04f1f3b38a412348f18d73b5e3ceb7bf41de1f7e199e437a04ea1aa735cd2f7b"
Oct 30 14:12:53.078743 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:53.078674    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04f1f3b38a412348f18d73b5e3ceb7bf41de1f7e199e437a04ea1aa735cd2f7b"} err="failed to get container status \"04f1f3b38a412348f18d73b5e3ceb7bf41de1f7e199e437a04ea1aa735cd2f7b\": rpc error: code = NotFound desc = could not find container \"04f1f3b38a412348f18d73b5e3ceb7bf41de1f7e199e437a04ea1aa735cd2f7b\": container with ID starting with 04f1f3b38a412348f18d73b5e3ceb7bf41de1f7e199e437a04ea1aa735cd2f7b not found: ID does not exist"
Oct 30 14:12:53.158537 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:53.158496    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6"]
Oct 30 14:12:53.169531 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:53.169496    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-xwwp6"]
Oct 30 14:12:54.998465 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:12:54.998427    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e378bf44-e323-460f-9096-fd8835ad91b8" path="/var/lib/kubelet/pods/e378bf44-e323-460f-9096-fd8835ad91b8/volumes"
Oct 30 14:13:01.917331 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:01.917300    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jdj4_f93ee017-ab79-447c-90e0-5b85b71dba24/console-operator/1.log"
Oct 30 14:13:01.935653 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:01.935618    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tnm_3a675ac7-fbe1-449b-accf-bb0456ed0bd3/ovn-acl-logging/0.log"
Oct 30 14:13:01.942331 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:01.942303    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jdj4_f93ee017-ab79-447c-90e0-5b85b71dba24/console-operator/1.log"
Oct 30 14:13:01.948092 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:01.948059    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tnm_3a675ac7-fbe1-449b-accf-bb0456ed0bd3/ovn-acl-logging/0.log"
Oct 30 14:13:08.085273 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:08.085223    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw" podUID="ad37954e-1350-4f75-a63f-a0c0f29ae416" containerName="main" containerID="cri-o://eee06b34a6dfc140bceba2e50b20760552420543366306b80dd1580e7e56c366" gracePeriod=45
Oct 30 14:13:08.561559 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:08.556850    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw"
Oct 30 14:13:08.638487 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:08.638428    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad37954e-1350-4f75-a63f-a0c0f29ae416-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad37954e-1350-4f75-a63f-a0c0f29ae416-tls-certs\") pod \"ad37954e-1350-4f75-a63f-a0c0f29ae416\" (UID: \"ad37954e-1350-4f75-a63f-a0c0f29ae416\") "
Oct 30 14:13:08.638872 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:08.638821    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad37954e-1350-4f75-a63f-a0c0f29ae416-kube-api-access-qmzbf\" (UniqueName: \"kubernetes.io/projected/ad37954e-1350-4f75-a63f-a0c0f29ae416-kube-api-access-qmzbf\") pod \"ad37954e-1350-4f75-a63f-a0c0f29ae416\" (UID: \"ad37954e-1350-4f75-a63f-a0c0f29ae416\") "
Oct 30 14:13:08.639029 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:08.638990    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad37954e-1350-4f75-a63f-a0c0f29ae416-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad37954e-1350-4f75-a63f-a0c0f29ae416-model-cache\") pod \"ad37954e-1350-4f75-a63f-a0c0f29ae416\" (UID: \"ad37954e-1350-4f75-a63f-a0c0f29ae416\") "
Oct 30 14:13:08.639226 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:08.639163    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad37954e-1350-4f75-a63f-a0c0f29ae416-home\" (UniqueName: \"kubernetes.io/empty-dir/ad37954e-1350-4f75-a63f-a0c0f29ae416-home\") pod \"ad37954e-1350-4f75-a63f-a0c0f29ae416\" (UID: \"ad37954e-1350-4f75-a63f-a0c0f29ae416\") "
Oct 30 14:13:08.639384 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:08.639209    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad37954e-1350-4f75-a63f-a0c0f29ae416-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad37954e-1350-4f75-a63f-a0c0f29ae416-tmp-dir\") pod \"ad37954e-1350-4f75-a63f-a0c0f29ae416\" (UID: \"ad37954e-1350-4f75-a63f-a0c0f29ae416\") "
Oct 30 14:13:08.640191 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:08.639519    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad37954e-1350-4f75-a63f-a0c0f29ae416-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad37954e-1350-4f75-a63f-a0c0f29ae416-dshm\") pod \"ad37954e-1350-4f75-a63f-a0c0f29ae416\" (UID: \"ad37954e-1350-4f75-a63f-a0c0f29ae416\") "
Oct 30 14:13:08.640744 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:08.640705    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad37954e-1350-4f75-a63f-a0c0f29ae416-model-cache" pod "ad37954e-1350-4f75-a63f-a0c0f29ae416" (UID: "ad37954e-1350-4f75-a63f-a0c0f29ae416"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:08.640988 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:08.640964    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad37954e-1350-4f75-a63f-a0c0f29ae416-home" pod "ad37954e-1350-4f75-a63f-a0c0f29ae416" (UID: "ad37954e-1350-4f75-a63f-a0c0f29ae416"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:08.641233 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:08.641196    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad37954e-1350-4f75-a63f-a0c0f29ae416-tmp-dir" pod "ad37954e-1350-4f75-a63f-a0c0f29ae416" (UID: "ad37954e-1350-4f75-a63f-a0c0f29ae416"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:08.643232 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:08.643191    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad37954e-1350-4f75-a63f-a0c0f29ae416-dshm" pod "ad37954e-1350-4f75-a63f-a0c0f29ae416" (UID: "ad37954e-1350-4f75-a63f-a0c0f29ae416"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:08.643933 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:08.643909    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad37954e-1350-4f75-a63f-a0c0f29ae416-kube-api-access-qmzbf" pod "ad37954e-1350-4f75-a63f-a0c0f29ae416" (UID: "ad37954e-1350-4f75-a63f-a0c0f29ae416"). InnerVolumeSpecName "kube-api-access-qmzbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:13:08.644996 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:08.644943    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad37954e-1350-4f75-a63f-a0c0f29ae416-tls-certs" pod "ad37954e-1350-4f75-a63f-a0c0f29ae416" (UID: "ad37954e-1350-4f75-a63f-a0c0f29ae416"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:13:08.741171 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:08.741037    5093 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad37954e-1350-4f75-a63f-a0c0f29ae416-tls-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:13:08.741171 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:08.741086    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmzbf\" (UniqueName: \"kubernetes.io/projected/ad37954e-1350-4f75-a63f-a0c0f29ae416-kube-api-access-qmzbf\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:13:08.741171 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:08.741102    5093 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad37954e-1350-4f75-a63f-a0c0f29ae416-model-cache\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:13:08.741171 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:08.741117    5093 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad37954e-1350-4f75-a63f-a0c0f29ae416-home\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:13:08.741171 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:08.741171    5093 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad37954e-1350-4f75-a63f-a0c0f29ae416-tmp-dir\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:13:08.741462 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:08.741184    5093 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad37954e-1350-4f75-a63f-a0c0f29ae416-dshm\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:13:09.168002 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:09.167958    5093 generic.go:356] "Generic (PLEG): container finished" podID="ad37954e-1350-4f75-a63f-a0c0f29ae416" containerID="eee06b34a6dfc140bceba2e50b20760552420543366306b80dd1580e7e56c366" exitCode=0
Oct 30 14:13:09.168551 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:09.168013    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw"
Oct 30 14:13:09.168551 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:09.168039    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw" event={"ID":"ad37954e-1350-4f75-a63f-a0c0f29ae416","Type":"ContainerDied","Data":"eee06b34a6dfc140bceba2e50b20760552420543366306b80dd1580e7e56c366"}
Oct 30 14:13:09.168551 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:09.168095    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw" event={"ID":"ad37954e-1350-4f75-a63f-a0c0f29ae416","Type":"ContainerDied","Data":"24d2a0d50321fd7da922b39d6d28a9aae8fcf07e8734c9eda1a151c7372d8db6"}
Oct 30 14:13:09.168551 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:09.168122    5093 scope.go:122] "RemoveContainer" containerID="eee06b34a6dfc140bceba2e50b20760552420543366306b80dd1580e7e56c366"
Oct 30 14:13:09.179089 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:09.179059    5093 scope.go:122] "RemoveContainer" containerID="eee06b34a6dfc140bceba2e50b20760552420543366306b80dd1580e7e56c366"
Oct 30 14:13:09.179462 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:13:09.179438    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eee06b34a6dfc140bceba2e50b20760552420543366306b80dd1580e7e56c366\": container with ID starting with eee06b34a6dfc140bceba2e50b20760552420543366306b80dd1580e7e56c366 not found: ID does not exist" containerID="eee06b34a6dfc140bceba2e50b20760552420543366306b80dd1580e7e56c366"
Oct 30 14:13:09.179552 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:09.179474    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eee06b34a6dfc140bceba2e50b20760552420543366306b80dd1580e7e56c366"} err="failed to get container status \"eee06b34a6dfc140bceba2e50b20760552420543366306b80dd1580e7e56c366\": rpc error: code = NotFound desc = could not find container \"eee06b34a6dfc140bceba2e50b20760552420543366306b80dd1580e7e56c366\": container with ID starting with eee06b34a6dfc140bceba2e50b20760552420543366306b80dd1580e7e56c366 not found: ID does not exist"
Oct 30 14:13:09.226565 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:09.226519    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw"]
Oct 30 14:13:09.233573 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:09.233534    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-52af49/distinct-w1-52af49-kserve-6d595fc8c-88lfw"]
Oct 30 14:13:10.999563 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:10.999521    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad37954e-1350-4f75-a63f-a0c0f29ae416" path="/var/lib/kubelet/pods/ad37954e-1350-4f75-a63f-a0c0f29ae416/volumes"
Oct 30 14:13:19.321861 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:19.321820    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-52af49-7b74b47477-c9f8d"]
Oct 30 14:13:19.323817 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:19.323780    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-52af49-7b74b47477-c9f8d" podUID="00cf3b24-1389-4fd2-a8f2-39c6acf5916c" containerName="maas-api" containerID="cri-o://f2daf5262312c24efc8e1eed8fdb4181a9b30b15e45b7e7adc79ab6cc345cfca" gracePeriod=30
Oct 30 14:13:19.600550 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:19.600522    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-52af49-7b74b47477-c9f8d"
Oct 30 14:13:19.751421 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:19.751382    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00cf3b24-1389-4fd2-a8f2-39c6acf5916c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/00cf3b24-1389-4fd2-a8f2-39c6acf5916c-metrics-certs\") pod \"00cf3b24-1389-4fd2-a8f2-39c6acf5916c\" (UID: \"00cf3b24-1389-4fd2-a8f2-39c6acf5916c\") "
Oct 30 14:13:19.751647 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:19.751444    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00cf3b24-1389-4fd2-a8f2-39c6acf5916c-kube-api-access-n7swk\" (UniqueName: \"kubernetes.io/projected/00cf3b24-1389-4fd2-a8f2-39c6acf5916c-kube-api-access-n7swk\") pod \"00cf3b24-1389-4fd2-a8f2-39c6acf5916c\" (UID: \"00cf3b24-1389-4fd2-a8f2-39c6acf5916c\") "
Oct 30 14:13:19.751647 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:19.751511    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00cf3b24-1389-4fd2-a8f2-39c6acf5916c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00cf3b24-1389-4fd2-a8f2-39c6acf5916c-maas-api-tls\") pod \"00cf3b24-1389-4fd2-a8f2-39c6acf5916c\" (UID: \"00cf3b24-1389-4fd2-a8f2-39c6acf5916c\") "
Oct 30 14:13:19.751647 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:19.751566    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00cf3b24-1389-4fd2-a8f2-39c6acf5916c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00cf3b24-1389-4fd2-a8f2-39c6acf5916c-openshift-service-ca\") pod \"00cf3b24-1389-4fd2-a8f2-39c6acf5916c\" (UID: \"00cf3b24-1389-4fd2-a8f2-39c6acf5916c\") "
Oct 30 14:13:19.752024 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:19.751989    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00cf3b24-1389-4fd2-a8f2-39c6acf5916c-openshift-service-ca" pod "00cf3b24-1389-4fd2-a8f2-39c6acf5916c" (UID: "00cf3b24-1389-4fd2-a8f2-39c6acf5916c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:13:19.754069 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:19.753999    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00cf3b24-1389-4fd2-a8f2-39c6acf5916c-metrics-certs" pod "00cf3b24-1389-4fd2-a8f2-39c6acf5916c" (UID: "00cf3b24-1389-4fd2-a8f2-39c6acf5916c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:13:19.754069 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:19.754046    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00cf3b24-1389-4fd2-a8f2-39c6acf5916c-kube-api-access-n7swk" pod "00cf3b24-1389-4fd2-a8f2-39c6acf5916c" (UID: "00cf3b24-1389-4fd2-a8f2-39c6acf5916c"). InnerVolumeSpecName "kube-api-access-n7swk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:13:19.754619 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:19.754598    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00cf3b24-1389-4fd2-a8f2-39c6acf5916c-maas-api-tls" pod "00cf3b24-1389-4fd2-a8f2-39c6acf5916c" (UID: "00cf3b24-1389-4fd2-a8f2-39c6acf5916c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:13:19.853180 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:19.853064    5093 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00cf3b24-1389-4fd2-a8f2-39c6acf5916c-maas-api-tls\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:13:19.853180 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:19.853097    5093 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00cf3b24-1389-4fd2-a8f2-39c6acf5916c-openshift-service-ca\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:13:19.853180 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:19.853108    5093 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00cf3b24-1389-4fd2-a8f2-39c6acf5916c-metrics-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:13:19.853180 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:19.853117    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7swk\" (UniqueName: \"kubernetes.io/projected/00cf3b24-1389-4fd2-a8f2-39c6acf5916c-kube-api-access-n7swk\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:13:20.235463 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:20.235424    5093 generic.go:356] "Generic (PLEG): container finished" podID="00cf3b24-1389-4fd2-a8f2-39c6acf5916c" containerID="f2daf5262312c24efc8e1eed8fdb4181a9b30b15e45b7e7adc79ab6cc345cfca" exitCode=0
Oct 30 14:13:20.235656 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:20.235499    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-52af49-7b74b47477-c9f8d"
Oct 30 14:13:20.235656 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:20.235503    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-52af49-7b74b47477-c9f8d" event={"ID":"00cf3b24-1389-4fd2-a8f2-39c6acf5916c","Type":"ContainerDied","Data":"f2daf5262312c24efc8e1eed8fdb4181a9b30b15e45b7e7adc79ab6cc345cfca"}
Oct 30 14:13:20.235656 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:20.235627    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-52af49-7b74b47477-c9f8d" event={"ID":"00cf3b24-1389-4fd2-a8f2-39c6acf5916c","Type":"ContainerDied","Data":"4f5aa88897d9f3a5e3c044d65ebedecbc47906fb4f2c869576bdbb6fa5e4961c"}
Oct 30 14:13:20.235656 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:20.235649    5093 scope.go:122] "RemoveContainer" containerID="f2daf5262312c24efc8e1eed8fdb4181a9b30b15e45b7e7adc79ab6cc345cfca"
Oct 30 14:13:20.246055 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:20.246018    5093 scope.go:122] "RemoveContainer" containerID="f2daf5262312c24efc8e1eed8fdb4181a9b30b15e45b7e7adc79ab6cc345cfca"
Oct 30 14:13:20.246537 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:13:20.246505    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2daf5262312c24efc8e1eed8fdb4181a9b30b15e45b7e7adc79ab6cc345cfca\": container with ID starting with f2daf5262312c24efc8e1eed8fdb4181a9b30b15e45b7e7adc79ab6cc345cfca not found: ID does not exist" containerID="f2daf5262312c24efc8e1eed8fdb4181a9b30b15e45b7e7adc79ab6cc345cfca"
Oct 30 14:13:20.246652 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:20.246551    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2daf5262312c24efc8e1eed8fdb4181a9b30b15e45b7e7adc79ab6cc345cfca"} err="failed to get container status \"f2daf5262312c24efc8e1eed8fdb4181a9b30b15e45b7e7adc79ab6cc345cfca\": rpc error: code = NotFound desc = could not find container \"f2daf5262312c24efc8e1eed8fdb4181a9b30b15e45b7e7adc79ab6cc345cfca\": container with ID starting with f2daf5262312c24efc8e1eed8fdb4181a9b30b15e45b7e7adc79ab6cc345cfca not found: ID does not exist"
Oct 30 14:13:20.289238 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:20.289202    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-52af49-7b74b47477-c9f8d"]
Oct 30 14:13:20.302577 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:20.302545    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-52af49-7b74b47477-c9f8d"]
Oct 30 14:13:20.998163 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:20.998114    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00cf3b24-1389-4fd2-a8f2-39c6acf5916c" path="/var/lib/kubelet/pods/00cf3b24-1389-4fd2-a8f2-39c6acf5916c/volumes"
Oct 30 14:13:21.168730 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:13:21.168693    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 14:13:22.643875 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:22.643839    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25"]
Oct 30 14:13:26.963939 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:26.963897    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj"]
Oct 30 14:13:26.964501 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:26.964232    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj" podUID="f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4" containerName="istio-proxy" containerID="cri-o://95ed86f855f248c3cd33894964957b249b06254dd5892c320941bc46a8c1a29e" gracePeriod=30
Oct 30 14:13:28.337427 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:28.337389    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-ec41e5-7b9c4d99f-v87cn"]
Oct 30 14:13:32.224366 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:32.224335    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj"
Oct 30 14:13:32.269782 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:32.269683    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-istio-podinfo\") pod \"f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4\" (UID: \"f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4\") "
Oct 30 14:13:32.269782 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:32.269756    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-istio-token\" (UniqueName: \"kubernetes.io/projected/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-istio-token\") pod \"f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4\" (UID: \"f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4\") "
Oct 30 14:13:32.270049 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:32.269794    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-istio-data\") pod \"f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4\" (UID: \"f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4\") "
Oct 30 14:13:32.270049 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:32.269812    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-istio-envoy\") pod \"f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4\" (UID: \"f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4\") "
Oct 30 14:13:32.270049 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:32.269841    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-istiod-ca-cert\") pod \"f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4\" (UID: \"f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4\") "
Oct 30 14:13:32.270049 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:32.269872    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-workload-socket\") pod \"f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4\" (UID: \"f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4\") "
Oct 30 14:13:32.270049 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:32.269980    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-credential-socket\") pod \"f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4\" (UID: \"f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4\") "
Oct 30 14:13:32.270049 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:32.270016    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-kube-api-access-tbmdq\" (UniqueName: \"kubernetes.io/projected/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-kube-api-access-tbmdq\") pod \"f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4\" (UID: \"f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4\") "
Oct 30 14:13:32.270426 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:32.270143    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-workload-certs\") pod \"f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4\" (UID: \"f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4\") "
Oct 30 14:13:32.270426 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:32.270337    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-istiod-ca-cert" pod "f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4" (UID: "f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:13:32.270542 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:32.270462    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-workload-socket" pod "f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4" (UID: "f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:32.270542 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:32.270385    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-istio-data" pod "f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4" (UID: "f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:32.270542 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:32.270400    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-credential-socket" pod "f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4" (UID: "f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:32.270711 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:32.270605    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-workload-certs" pod "f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4" (UID: "f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:32.270711 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:32.270683    5093 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-istio-data\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:13:32.270711 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:32.270706    5093 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-istiod-ca-cert\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:13:32.270877 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:32.270720    5093 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-workload-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:13:32.270877 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:32.270733    5093 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-credential-socket\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:13:32.272457 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:32.272427    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-istio-envoy" pod "f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4" (UID: "f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:32.272807 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:32.272775    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-istio-podinfo" pod "f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4" (UID: "f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:13:32.272954 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:32.272865    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-kube-api-access-tbmdq" pod "f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4" (UID: "f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4"). InnerVolumeSpecName "kube-api-access-tbmdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:13:32.273106 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:32.273087    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-istio-token" pod "f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4" (UID: "f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:13:32.311769 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:32.311734    5093 generic.go:356] "Generic (PLEG): container finished" podID="f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4" containerID="95ed86f855f248c3cd33894964957b249b06254dd5892c320941bc46a8c1a29e" exitCode=0
Oct 30 14:13:32.311963 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:32.311792    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj"
Oct 30 14:13:32.311963 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:32.311824    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj" event={"ID":"f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4","Type":"ContainerDied","Data":"95ed86f855f248c3cd33894964957b249b06254dd5892c320941bc46a8c1a29e"}
Oct 30 14:13:32.311963 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:32.311887    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj" event={"ID":"f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4","Type":"ContainerDied","Data":"002a7dd3a4c32ac93e4eeed42007ae5f26d53ee4f9ac82662b5a4a08e6b2c1bc"}
Oct 30 14:13:32.311963 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:32.311919    5093 scope.go:122] "RemoveContainer" containerID="95ed86f855f248c3cd33894964957b249b06254dd5892c320941bc46a8c1a29e"
Oct 30 14:13:32.325229 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:32.325203    5093 scope.go:122] "RemoveContainer" containerID="95ed86f855f248c3cd33894964957b249b06254dd5892c320941bc46a8c1a29e"
Oct 30 14:13:32.325618 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:13:32.325589    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95ed86f855f248c3cd33894964957b249b06254dd5892c320941bc46a8c1a29e\": container with ID starting with 95ed86f855f248c3cd33894964957b249b06254dd5892c320941bc46a8c1a29e not found: ID does not exist" containerID="95ed86f855f248c3cd33894964957b249b06254dd5892c320941bc46a8c1a29e"
Oct 30 14:13:32.325705 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:32.325634    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95ed86f855f248c3cd33894964957b249b06254dd5892c320941bc46a8c1a29e"} err="failed to get container status \"95ed86f855f248c3cd33894964957b249b06254dd5892c320941bc46a8c1a29e\": rpc error: code = NotFound desc = could not find container \"95ed86f855f248c3cd33894964957b249b06254dd5892c320941bc46a8c1a29e\": container with ID starting with 95ed86f855f248c3cd33894964957b249b06254dd5892c320941bc46a8c1a29e not found: ID does not exist"
Oct 30 14:13:32.366386 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:32.366345    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj"]
Oct 30 14:13:32.371423 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:32.371395    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbmdq\" (UniqueName: \"kubernetes.io/projected/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-kube-api-access-tbmdq\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:13:32.371557 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:32.371426    5093 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-workload-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:13:32.371557 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:32.371443    5093 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-istio-podinfo\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:13:32.371557 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:32.371453    5093 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-istio-token\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:13:32.371557 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:32.371460    5093 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4-istio-envoy\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:13:32.377240 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:32.377211    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-5gtgj"]
Oct 30 14:13:32.999911 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:32.999867    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4" path="/var/lib/kubelet/pods/f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4/volumes"
Oct 30 14:13:33.338673 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:33.338576    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ec41e5-7b9c4d99f-v87cn" podUID="7c86ceb7-5877-4c9a-988a-69e8217e6e08" containerName="payload-pre-processing" containerID="cri-o://6c88d7e092420ecd7dde71bc7b57852f68771ff12b3cbd5946ce3b9241cadfd5" gracePeriod=25
Oct 30 14:13:35.663689 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:35.663648    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ec41e5-7b9c4d99f-v87cn" podUID="7c86ceb7-5877-4c9a-988a-69e8217e6e08" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:13:37.667081 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:37.667017    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25" podUID="7805f89f-89d4-4754-b606-c3070f4df215" containerName="main" containerID="cri-o://3275d31c19ab423200a8e6cc380046b4de705485c42a5c0fb111eded99e1d24a" gracePeriod=45
Oct 30 14:13:37.923353 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:37.923323    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25"
Oct 30 14:13:38.030972 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:38.030933    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7805f89f-89d4-4754-b606-c3070f4df215-tls-certs\" (UniqueName: \"kubernetes.io/secret/7805f89f-89d4-4754-b606-c3070f4df215-tls-certs\") pod \"7805f89f-89d4-4754-b606-c3070f4df215\" (UID: \"7805f89f-89d4-4754-b606-c3070f4df215\") "
Oct 30 14:13:38.031191 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:38.031048    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7805f89f-89d4-4754-b606-c3070f4df215-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7805f89f-89d4-4754-b606-c3070f4df215-model-cache\") pod \"7805f89f-89d4-4754-b606-c3070f4df215\" (UID: \"7805f89f-89d4-4754-b606-c3070f4df215\") "
Oct 30 14:13:38.031191 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:38.031098    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7805f89f-89d4-4754-b606-c3070f4df215-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7805f89f-89d4-4754-b606-c3070f4df215-tmp-dir\") pod \"7805f89f-89d4-4754-b606-c3070f4df215\" (UID: \"7805f89f-89d4-4754-b606-c3070f4df215\") "
Oct 30 14:13:38.031191 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:38.031177    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7805f89f-89d4-4754-b606-c3070f4df215-kube-api-access-94pw6\" (UniqueName: \"kubernetes.io/projected/7805f89f-89d4-4754-b606-c3070f4df215-kube-api-access-94pw6\") pod \"7805f89f-89d4-4754-b606-c3070f4df215\" (UID: \"7805f89f-89d4-4754-b606-c3070f4df215\") "
Oct 30 14:13:38.031326 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:38.031227    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7805f89f-89d4-4754-b606-c3070f4df215-home\" (UniqueName: \"kubernetes.io/empty-dir/7805f89f-89d4-4754-b606-c3070f4df215-home\") pod \"7805f89f-89d4-4754-b606-c3070f4df215\" (UID: \"7805f89f-89d4-4754-b606-c3070f4df215\") "
Oct 30 14:13:38.031326 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:38.031263    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7805f89f-89d4-4754-b606-c3070f4df215-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7805f89f-89d4-4754-b606-c3070f4df215-dshm\") pod \"7805f89f-89d4-4754-b606-c3070f4df215\" (UID: \"7805f89f-89d4-4754-b606-c3070f4df215\") "
Oct 30 14:13:38.032110 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:38.031742    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7805f89f-89d4-4754-b606-c3070f4df215-home" pod "7805f89f-89d4-4754-b606-c3070f4df215" (UID: "7805f89f-89d4-4754-b606-c3070f4df215"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:38.032110 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:38.031910    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7805f89f-89d4-4754-b606-c3070f4df215-tmp-dir" pod "7805f89f-89d4-4754-b606-c3070f4df215" (UID: "7805f89f-89d4-4754-b606-c3070f4df215"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:38.032110 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:38.032082    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7805f89f-89d4-4754-b606-c3070f4df215-model-cache" pod "7805f89f-89d4-4754-b606-c3070f4df215" (UID: "7805f89f-89d4-4754-b606-c3070f4df215"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:38.034450 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:38.034420    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7805f89f-89d4-4754-b606-c3070f4df215-kube-api-access-94pw6" pod "7805f89f-89d4-4754-b606-c3070f4df215" (UID: "7805f89f-89d4-4754-b606-c3070f4df215"). InnerVolumeSpecName "kube-api-access-94pw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:13:38.034585 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:38.034557    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7805f89f-89d4-4754-b606-c3070f4df215-tls-certs" pod "7805f89f-89d4-4754-b606-c3070f4df215" (UID: "7805f89f-89d4-4754-b606-c3070f4df215"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:13:38.037583 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:38.037536    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7805f89f-89d4-4754-b606-c3070f4df215-dshm" pod "7805f89f-89d4-4754-b606-c3070f4df215" (UID: "7805f89f-89d4-4754-b606-c3070f4df215"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:38.132614 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:38.132571    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94pw6\" (UniqueName: \"kubernetes.io/projected/7805f89f-89d4-4754-b606-c3070f4df215-kube-api-access-94pw6\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:13:38.132614 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:38.132606    5093 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7805f89f-89d4-4754-b606-c3070f4df215-home\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:13:38.132614 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:38.132616    5093 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7805f89f-89d4-4754-b606-c3070f4df215-dshm\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:13:38.132614 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:38.132624    5093 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7805f89f-89d4-4754-b606-c3070f4df215-tls-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:13:38.133098 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:38.132633    5093 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7805f89f-89d4-4754-b606-c3070f4df215-model-cache\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:13:38.133098 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:38.132641    5093 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7805f89f-89d4-4754-b606-c3070f4df215-tmp-dir\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:13:38.350114 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:38.350002    5093 generic.go:356] "Generic (PLEG): container finished" podID="7805f89f-89d4-4754-b606-c3070f4df215" containerID="3275d31c19ab423200a8e6cc380046b4de705485c42a5c0fb111eded99e1d24a" exitCode=0
Oct 30 14:13:38.350114 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:38.350066    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25"
Oct 30 14:13:38.350114 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:38.350082    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25" event={"ID":"7805f89f-89d4-4754-b606-c3070f4df215","Type":"ContainerDied","Data":"3275d31c19ab423200a8e6cc380046b4de705485c42a5c0fb111eded99e1d24a"}
Oct 30 14:13:38.350421 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:38.350148    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25" event={"ID":"7805f89f-89d4-4754-b606-c3070f4df215","Type":"ContainerDied","Data":"248f496f2632a99009e1d51baa789c277b32699fa36d432626c595ac5c7587f4"}
Oct 30 14:13:38.350421 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:38.350170    5093 scope.go:122] "RemoveContainer" containerID="3275d31c19ab423200a8e6cc380046b4de705485c42a5c0fb111eded99e1d24a"
Oct 30 14:13:38.362341 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:38.362319    5093 scope.go:122] "RemoveContainer" containerID="3275d31c19ab423200a8e6cc380046b4de705485c42a5c0fb111eded99e1d24a"
Oct 30 14:13:38.362840 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:13:38.362810    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3275d31c19ab423200a8e6cc380046b4de705485c42a5c0fb111eded99e1d24a\": container with ID starting with 3275d31c19ab423200a8e6cc380046b4de705485c42a5c0fb111eded99e1d24a not found: ID does not exist" containerID="3275d31c19ab423200a8e6cc380046b4de705485c42a5c0fb111eded99e1d24a"
Oct 30 14:13:38.362945 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:38.362849    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3275d31c19ab423200a8e6cc380046b4de705485c42a5c0fb111eded99e1d24a"} err="failed to get container status \"3275d31c19ab423200a8e6cc380046b4de705485c42a5c0fb111eded99e1d24a\": rpc error: code = NotFound desc = could not find container \"3275d31c19ab423200a8e6cc380046b4de705485c42a5c0fb111eded99e1d24a\": container with ID starting with 3275d31c19ab423200a8e6cc380046b4de705485c42a5c0fb111eded99e1d24a not found: ID does not exist"
Oct 30 14:13:38.404768 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:38.404732    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25"]
Oct 30 14:13:38.411212 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:38.411184    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-ec41e5/distinct-2-w2-ec41e5-kserve-cfc76986f-f4r25"]
Oct 30 14:13:38.997576 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:38.997539    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7805f89f-89d4-4754-b606-c3070f4df215" path="/var/lib/kubelet/pods/7805f89f-89d4-4754-b606-c3070f4df215/volumes"
Oct 30 14:13:44.814257 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:44.814208    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-ec41e5-565987664-cg77b"]
Oct 30 14:13:44.814742 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:44.814480    5093 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ec41e5-565987664-cg77b" podUID="7c30da13-27d5-4947-82e1-21ef1aba9460" containerName="maas-api" containerID="cri-o://8f8fc835ed83a43ef9f9518b3d65a29267a8e76dc5e5ba32fed9f4768915f468" gracePeriod=30
Oct 30 14:13:45.086361 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:45.086330    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ec41e5-565987664-cg77b"
Oct 30 14:13:45.200143 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:45.200096    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c30da13-27d5-4947-82e1-21ef1aba9460-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c30da13-27d5-4947-82e1-21ef1aba9460-metrics-certs\") pod \"7c30da13-27d5-4947-82e1-21ef1aba9460\" (UID: \"7c30da13-27d5-4947-82e1-21ef1aba9460\") "
Oct 30 14:13:45.200356 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:45.200183    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c30da13-27d5-4947-82e1-21ef1aba9460-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c30da13-27d5-4947-82e1-21ef1aba9460-maas-api-tls\") pod \"7c30da13-27d5-4947-82e1-21ef1aba9460\" (UID: \"7c30da13-27d5-4947-82e1-21ef1aba9460\") "
Oct 30 14:13:45.200356 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:45.200289    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c30da13-27d5-4947-82e1-21ef1aba9460-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c30da13-27d5-4947-82e1-21ef1aba9460-openshift-service-ca\") pod \"7c30da13-27d5-4947-82e1-21ef1aba9460\" (UID: \"7c30da13-27d5-4947-82e1-21ef1aba9460\") "
Oct 30 14:13:45.200356 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:45.200342    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c30da13-27d5-4947-82e1-21ef1aba9460-kube-api-access-jzx7w\" (UniqueName: \"kubernetes.io/projected/7c30da13-27d5-4947-82e1-21ef1aba9460-kube-api-access-jzx7w\") pod \"7c30da13-27d5-4947-82e1-21ef1aba9460\" (UID: \"7c30da13-27d5-4947-82e1-21ef1aba9460\") "
Oct 30 14:13:45.200695 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:45.200663    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c30da13-27d5-4947-82e1-21ef1aba9460-openshift-service-ca" pod "7c30da13-27d5-4947-82e1-21ef1aba9460" (UID: "7c30da13-27d5-4947-82e1-21ef1aba9460"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:13:45.202423 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:45.202392    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c30da13-27d5-4947-82e1-21ef1aba9460-metrics-certs" pod "7c30da13-27d5-4947-82e1-21ef1aba9460" (UID: "7c30da13-27d5-4947-82e1-21ef1aba9460"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:13:45.202549 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:45.202527    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c30da13-27d5-4947-82e1-21ef1aba9460-maas-api-tls" pod "7c30da13-27d5-4947-82e1-21ef1aba9460" (UID: "7c30da13-27d5-4947-82e1-21ef1aba9460"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:13:45.202618 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:45.202550    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c30da13-27d5-4947-82e1-21ef1aba9460-kube-api-access-jzx7w" pod "7c30da13-27d5-4947-82e1-21ef1aba9460" (UID: "7c30da13-27d5-4947-82e1-21ef1aba9460"). InnerVolumeSpecName "kube-api-access-jzx7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:13:45.301561 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:45.301518    5093 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c30da13-27d5-4947-82e1-21ef1aba9460-metrics-certs\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:13:45.301561 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:45.301556    5093 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c30da13-27d5-4947-82e1-21ef1aba9460-maas-api-tls\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:13:45.301795 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:45.301572    5093 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c30da13-27d5-4947-82e1-21ef1aba9460-openshift-service-ca\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:13:45.301795 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:45.301586    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzx7w\" (UniqueName: \"kubernetes.io/projected/7c30da13-27d5-4947-82e1-21ef1aba9460-kube-api-access-jzx7w\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:13:45.393482 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:45.393393    5093 generic.go:356] "Generic (PLEG): container finished" podID="7c30da13-27d5-4947-82e1-21ef1aba9460" containerID="8f8fc835ed83a43ef9f9518b3d65a29267a8e76dc5e5ba32fed9f4768915f468" exitCode=0
Oct 30 14:13:45.393482 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:45.393440    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ec41e5-565987664-cg77b"
Oct 30 14:13:45.393482 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:45.393467    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ec41e5-565987664-cg77b" event={"ID":"7c30da13-27d5-4947-82e1-21ef1aba9460","Type":"ContainerDied","Data":"8f8fc835ed83a43ef9f9518b3d65a29267a8e76dc5e5ba32fed9f4768915f468"}
Oct 30 14:13:45.393740 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:45.393527    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ec41e5-565987664-cg77b" event={"ID":"7c30da13-27d5-4947-82e1-21ef1aba9460","Type":"ContainerDied","Data":"183b74f6c59298cf17d9791b1d559b37ebea13916ad6039ff4ca3e0adf332137"}
Oct 30 14:13:45.393740 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:45.393554    5093 scope.go:122] "RemoveContainer" containerID="8f8fc835ed83a43ef9f9518b3d65a29267a8e76dc5e5ba32fed9f4768915f468"
Oct 30 14:13:45.404814 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:45.404787    5093 scope.go:122] "RemoveContainer" containerID="8f8fc835ed83a43ef9f9518b3d65a29267a8e76dc5e5ba32fed9f4768915f468"
Oct 30 14:13:45.405198 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:13:45.405173    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f8fc835ed83a43ef9f9518b3d65a29267a8e76dc5e5ba32fed9f4768915f468\": container with ID starting with 8f8fc835ed83a43ef9f9518b3d65a29267a8e76dc5e5ba32fed9f4768915f468 not found: ID does not exist" containerID="8f8fc835ed83a43ef9f9518b3d65a29267a8e76dc5e5ba32fed9f4768915f468"
Oct 30 14:13:45.405290 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:45.405214    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f8fc835ed83a43ef9f9518b3d65a29267a8e76dc5e5ba32fed9f4768915f468"} err="failed to get container status \"8f8fc835ed83a43ef9f9518b3d65a29267a8e76dc5e5ba32fed9f4768915f468\": rpc error: code = NotFound desc = could not find container \"8f8fc835ed83a43ef9f9518b3d65a29267a8e76dc5e5ba32fed9f4768915f468\": container with ID starting with 8f8fc835ed83a43ef9f9518b3d65a29267a8e76dc5e5ba32fed9f4768915f468 not found: ID does not exist"
Oct 30 14:13:45.447313 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:45.447274    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-ec41e5-565987664-cg77b"]
Oct 30 14:13:45.451284 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:45.451253    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-ec41e5-565987664-cg77b"]
Oct 30 14:13:45.664168 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:45.664056    5093 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ec41e5-7b9c4d99f-v87cn" podUID="7c86ceb7-5877-4c9a-988a-69e8217e6e08" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:13:46.998274 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:46.998238    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c30da13-27d5-4947-82e1-21ef1aba9460" path="/var/lib/kubelet/pods/7c30da13-27d5-4947-82e1-21ef1aba9460/volumes"
Oct 30 14:13:51.168404 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:13:51.168361    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 14:13:53.601465 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:53.601399    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ec41e5-7b9c4d99f-v87cn"
Oct 30 14:13:53.683292 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:53.683250    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c86ceb7-5877-4c9a-988a-69e8217e6e08-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c86ceb7-5877-4c9a-988a-69e8217e6e08-plugins-config-volume\") pod \"7c86ceb7-5877-4c9a-988a-69e8217e6e08\" (UID: \"7c86ceb7-5877-4c9a-988a-69e8217e6e08\") "
Oct 30 14:13:53.683484 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:53.683322    5093 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c86ceb7-5877-4c9a-988a-69e8217e6e08-kube-api-access-w6gpp\" (UniqueName: \"kubernetes.io/projected/7c86ceb7-5877-4c9a-988a-69e8217e6e08-kube-api-access-w6gpp\") pod \"7c86ceb7-5877-4c9a-988a-69e8217e6e08\" (UID: \"7c86ceb7-5877-4c9a-988a-69e8217e6e08\") "
Oct 30 14:13:53.683672 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:53.683648    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c86ceb7-5877-4c9a-988a-69e8217e6e08-plugins-config-volume" pod "7c86ceb7-5877-4c9a-988a-69e8217e6e08" (UID: "7c86ceb7-5877-4c9a-988a-69e8217e6e08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:13:53.685674 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:53.685638    5093 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c86ceb7-5877-4c9a-988a-69e8217e6e08-kube-api-access-w6gpp" pod "7c86ceb7-5877-4c9a-988a-69e8217e6e08" (UID: "7c86ceb7-5877-4c9a-988a-69e8217e6e08"). InnerVolumeSpecName "kube-api-access-w6gpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:13:53.784610 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:53.784566    5093 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c86ceb7-5877-4c9a-988a-69e8217e6e08-plugins-config-volume\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:13:53.784610 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:53.784604    5093 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6gpp\" (UniqueName: \"kubernetes.io/projected/7c86ceb7-5877-4c9a-988a-69e8217e6e08-kube-api-access-w6gpp\") on node \"ip-10-0-134-31.ec2.internal\" DevicePath \"\""
Oct 30 14:13:54.229820 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:54.229785    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-568c86c4df-qqzql_a6cde889-a80c-470e-86ea-1d930042a4ef/authorino/0.log"
Oct 30 14:13:54.448671 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:54.448633    5093 generic.go:356] "Generic (PLEG): container finished" podID="7c86ceb7-5877-4c9a-988a-69e8217e6e08" containerID="6c88d7e092420ecd7dde71bc7b57852f68771ff12b3cbd5946ce3b9241cadfd5" exitCode=0
Oct 30 14:13:54.448869 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:54.448684    5093 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ec41e5-7b9c4d99f-v87cn"
Oct 30 14:13:54.448869 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:54.448723    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ec41e5-7b9c4d99f-v87cn" event={"ID":"7c86ceb7-5877-4c9a-988a-69e8217e6e08","Type":"ContainerDied","Data":"6c88d7e092420ecd7dde71bc7b57852f68771ff12b3cbd5946ce3b9241cadfd5"}
Oct 30 14:13:54.448869 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:54.448783    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ec41e5-7b9c4d99f-v87cn" event={"ID":"7c86ceb7-5877-4c9a-988a-69e8217e6e08","Type":"ContainerDied","Data":"889764dd2a62df14eaf9211199661d541279b4fd56a288caddb5517acf176912"}
Oct 30 14:13:54.448869 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:54.448813    5093 scope.go:122] "RemoveContainer" containerID="6c88d7e092420ecd7dde71bc7b57852f68771ff12b3cbd5946ce3b9241cadfd5"
Oct 30 14:13:54.459536 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:54.459313    5093 scope.go:122] "RemoveContainer" containerID="6c88d7e092420ecd7dde71bc7b57852f68771ff12b3cbd5946ce3b9241cadfd5"
Oct 30 14:13:54.459609 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:13:54.459586    5093 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c88d7e092420ecd7dde71bc7b57852f68771ff12b3cbd5946ce3b9241cadfd5\": container with ID starting with 6c88d7e092420ecd7dde71bc7b57852f68771ff12b3cbd5946ce3b9241cadfd5 not found: ID does not exist" containerID="6c88d7e092420ecd7dde71bc7b57852f68771ff12b3cbd5946ce3b9241cadfd5"
Oct 30 14:13:54.459669 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:54.459621    5093 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c88d7e092420ecd7dde71bc7b57852f68771ff12b3cbd5946ce3b9241cadfd5"} err="failed to get container status \"6c88d7e092420ecd7dde71bc7b57852f68771ff12b3cbd5946ce3b9241cadfd5\": rpc error: code = NotFound desc = could not find container \"6c88d7e092420ecd7dde71bc7b57852f68771ff12b3cbd5946ce3b9241cadfd5\": container with ID starting with 6c88d7e092420ecd7dde71bc7b57852f68771ff12b3cbd5946ce3b9241cadfd5 not found: ID does not exist"
Oct 30 14:13:54.478082 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:54.478038    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-ntjsh_8b8d6a8f-a170-4d6d-82f2-1d72828b55c5/manager/0.log"
Oct 30 14:13:54.502757 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:54.502658    5093 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-ec41e5-7b9c4d99f-v87cn"]
Oct 30 14:13:54.515286 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:54.515250    5093 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-ec41e5-7b9c4d99f-v87cn"]
Oct 30 14:13:54.687806 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:54.687766    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-p4trh_a955c0a1-e0ce-4ee8-8320-345bd556eff6/manager/0.log"
Oct 30 14:13:54.998552 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:54.998513    5093 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c86ceb7-5877-4c9a-988a-69e8217e6e08" path="/var/lib/kubelet/pods/7c86ceb7-5877-4c9a-988a-69e8217e6e08/volumes"
Oct 30 14:13:55.399668 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:55.399571    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-zfv7q_fc701686-ddfa-4eca-be79-b8790e56d681/manager/0.log"
Oct 30 14:13:55.823445 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:55.823407    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-2fcf7_ad961809-9f3f-43dd-b97a-c4d6ebaa4457/manager/0.log"
Oct 30 14:13:58.007457 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:58.007423    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-k8hb7_ebddb579-1088-4d8f-8917-e7e14883534a/server/0.log"
Oct 30 14:13:58.197523 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:58.197486    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-hdnsr_775251e0-43d1-4239-a077-1b9fc7e56fe4/manager/0.log"
Oct 30 14:13:59.406866 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:13:59.406829    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-568c86c4df-qqzql_a6cde889-a80c-470e-86ea-1d930042a4ef/authorino/0.log"
Oct 30 14:14:11.427438 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:11.427354    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-k8hb7_ebddb579-1088-4d8f-8917-e7e14883534a/server/0.log"
Oct 30 14:14:11.621204 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:11.621165    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-hdnsr_775251e0-43d1-4239-a077-1b9fc7e56fe4/manager/0.log"
Oct 30 14:14:14.305462 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:14.305433    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-78ac3eda-hcrzl_c8f349f4-3e49-4c63-bc17-1d220cb7f477/revoke-keys/0.log"
Oct 30 14:14:17.551701 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:17.551665    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-568c86c4df-qqzql_a6cde889-a80c-470e-86ea-1d930042a4ef/authorino/0.log"
Oct 30 14:14:17.793909 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:17.793873    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-ntjsh_8b8d6a8f-a170-4d6d-82f2-1d72828b55c5/manager/0.log"
Oct 30 14:14:17.985454 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:17.985414    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-p4trh_a955c0a1-e0ce-4ee8-8320-345bd556eff6/manager/0.log"
Oct 30 14:14:18.583731 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:18.583693    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-zfv7q_fc701686-ddfa-4eca-be79-b8790e56d681/manager/0.log"
Oct 30 14:14:18.979148 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:18.979093    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-2fcf7_ad961809-9f3f-43dd-b97a-c4d6ebaa4457/manager/0.log"
Oct 30 14:14:19.732213 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:19.732175    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l_142560f0-3d00-4d0c-9f87-58746dbe8a4e/istio-proxy/0.log"
Oct 30 14:14:21.167512 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:14:21.167474    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 14:14:21.898773 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:21.898739    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-668b54766c-wgk28_d5db3ba8-f1de-4fa5-9839-e7e8200da58f/router/0.log"
Oct 30 14:14:23.057346 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:23.057303    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-z5nd6_47d931a7-02ab-412b-a753-6bead66a76cd/main/0.log"
Oct 30 14:14:23.455248 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:23.455217    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b867hfz_1007aa25-0492-47a8-b0d5-a864fec1022a/main/0.log"
Oct 30 14:14:23.860818 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:23.860735    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-7hplr_51cafd88-c303-4185-ac30-aa9f20b1ed4f/main/0.log"
Oct 30 14:14:24.064521 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:24.064491    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-jtshg_ea079ac1-f502-4b5f-b16a-c91e2de98de8/main/0.log"
Oct 30 14:14:40.198583 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:40.198547    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wrfqz_10b774a1-1476-4f91-b6e5-d52b3f45700e/global-pull-secret-syncer/0.log"
Oct 30 14:14:40.380657 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:40.380630    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-22fm4_46bedd0c-7237-4128-be98-69e4474bb535/connection-checker/0.log"
Oct 30 14:14:40.525930 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:40.525848    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-rbf7t_8021bc0b-4447-4f04-b9bd-5f80e9cf3507/connection-checker/0.log"
Oct 30 14:14:40.676303 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:40.676267    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-wnzh2_44f513ab-9115-4761-aa8b-c546212ec17b/connection-checker/0.log"
Oct 30 14:14:41.027849 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:41.027813    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wpqvv_beb2b7e4-b43d-42fc-9e8a-ecdfa6994205/konnectivity-agent/0.log"
Oct 30 14:14:41.447108 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:41.447076    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-31.ec2.internal_5213eb0d1df39bb19f1777f20f292baf/haproxy/0.log"
Oct 30 14:14:48.774941 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:48.774899    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-568c86c4df-qqzql_a6cde889-a80c-470e-86ea-1d930042a4ef/authorino/0.log"
Oct 30 14:14:48.893498 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:48.893463    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-ntjsh_8b8d6a8f-a170-4d6d-82f2-1d72828b55c5/manager/0.log"
Oct 30 14:14:49.024427 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:49.024394    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-p4trh_a955c0a1-e0ce-4ee8-8320-345bd556eff6/manager/0.log"
Oct 30 14:14:49.625937 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:49.625892    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-zfv7q_fc701686-ddfa-4eca-be79-b8790e56d681/manager/0.log"
Oct 30 14:14:50.427202 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:50.427148    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-2fcf7_ad961809-9f3f-43dd-b97a-c4d6ebaa4457/manager/0.log"
Oct 30 14:14:51.165952 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:14:51.165911    5093 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nk2zz" containerName="extractor"
Oct 30 14:14:51.471723 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.471685    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fhsqk/perf-node-gather-daemonset-7qnk2"]
Oct 30 14:14:51.472153 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.472100    5093 cpu_manager.go:414] "containerMap: removing container" podUID="c8f349f4-3e49-4c63-bc17-1d220cb7f477" containerName="revoke-keys"
Oct 30 14:14:51.472153 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.472111    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8f349f4-3e49-4c63-bc17-1d220cb7f477" containerName="revoke-keys"
Oct 30 14:14:51.472153 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.472122    5093 cpu_manager.go:414] "containerMap: removing container" podUID="e378bf44-e323-460f-9096-fd8835ad91b8" containerName="istio-proxy"
Oct 30 14:14:51.472153 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.472140    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e378bf44-e323-460f-9096-fd8835ad91b8" containerName="istio-proxy"
Oct 30 14:14:51.472153 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.472154    5093 cpu_manager.go:414] "containerMap: removing container" podUID="00cf3b24-1389-4fd2-a8f2-39c6acf5916c" containerName="maas-api"
Oct 30 14:14:51.472366 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.472160    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00cf3b24-1389-4fd2-a8f2-39c6acf5916c" containerName="maas-api"
Oct 30 14:14:51.472366 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.472171    5093 cpu_manager.go:414] "containerMap: removing container" podUID="b0d3d9d3-593a-478d-963b-d8b4b395a661" containerName="istio-proxy"
Oct 30 14:14:51.472366 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.472176    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0d3d9d3-593a-478d-963b-d8b4b395a661" containerName="istio-proxy"
Oct 30 14:14:51.472366 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.472183    5093 cpu_manager.go:414] "containerMap: removing container" podUID="7c30da13-27d5-4947-82e1-21ef1aba9460" containerName="maas-api"
Oct 30 14:14:51.472366 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.472188    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c30da13-27d5-4947-82e1-21ef1aba9460" containerName="maas-api"
Oct 30 14:14:51.472366 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.472194    5093 cpu_manager.go:414] "containerMap: removing container" podUID="f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4" containerName="istio-proxy"
Oct 30 14:14:51.472366 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.472198    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4" containerName="istio-proxy"
Oct 30 14:14:51.472366 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.472205    5093 cpu_manager.go:414] "containerMap: removing container" podUID="ad37954e-1350-4f75-a63f-a0c0f29ae416" containerName="main"
Oct 30 14:14:51.472366 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.472209    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad37954e-1350-4f75-a63f-a0c0f29ae416" containerName="main"
Oct 30 14:14:51.472366 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.472217    5093 cpu_manager.go:414] "containerMap: removing container" podUID="ec7e602c-fb95-46e6-bbaa-74dd09dbebe5" containerName="maas-api"
Oct 30 14:14:51.472366 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.472224    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec7e602c-fb95-46e6-bbaa-74dd09dbebe5" containerName="maas-api"
Oct 30 14:14:51.472366 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.472236    5093 cpu_manager.go:414] "containerMap: removing container" podUID="7805f89f-89d4-4754-b606-c3070f4df215" containerName="main"
Oct 30 14:14:51.472366 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.472241    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7805f89f-89d4-4754-b606-c3070f4df215" containerName="main"
Oct 30 14:14:51.472366 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.472250    5093 cpu_manager.go:414] "containerMap: removing container" podUID="8285d6ec-8ad7-4a33-a696-d32242da2f80" containerName="istio-proxy"
Oct 30 14:14:51.472366 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.472254    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8285d6ec-8ad7-4a33-a696-d32242da2f80" containerName="istio-proxy"
Oct 30 14:14:51.472366 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.472261    5093 cpu_manager.go:414] "containerMap: removing container" podUID="7c86ceb7-5877-4c9a-988a-69e8217e6e08" containerName="payload-pre-processing"
Oct 30 14:14:51.472366 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.472265    5093 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c86ceb7-5877-4c9a-988a-69e8217e6e08" containerName="payload-pre-processing"
Oct 30 14:14:51.472366 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.472327    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8f349f4-3e49-4c63-bc17-1d220cb7f477" containerName="revoke-keys"
Oct 30 14:14:51.472366 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.472336    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="e378bf44-e323-460f-9096-fd8835ad91b8" containerName="istio-proxy"
Oct 30 14:14:51.472366 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.472343    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="00cf3b24-1389-4fd2-a8f2-39c6acf5916c" containerName="maas-api"
Oct 30 14:14:51.472366 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.472350    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3c6ad8f-cf78-4dfb-8a99-a9947fe676f4" containerName="istio-proxy"
Oct 30 14:14:51.472366 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.472356    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c30da13-27d5-4947-82e1-21ef1aba9460" containerName="maas-api"
Oct 30 14:14:51.472366 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.472361    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad37954e-1350-4f75-a63f-a0c0f29ae416" containerName="main"
Oct 30 14:14:51.472366 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.472367    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec7e602c-fb95-46e6-bbaa-74dd09dbebe5" containerName="maas-api"
Oct 30 14:14:51.472366 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.472373    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="7805f89f-89d4-4754-b606-c3070f4df215" containerName="main"
Oct 30 14:14:51.472366 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.472379    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c86ceb7-5877-4c9a-988a-69e8217e6e08" containerName="payload-pre-processing"
Oct 30 14:14:51.472366 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.472388    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="8285d6ec-8ad7-4a33-a696-d32242da2f80" containerName="istio-proxy"
Oct 30 14:14:51.472366 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.472394    5093 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0d3d9d3-593a-478d-963b-d8b4b395a661" containerName="istio-proxy"
Oct 30 14:14:51.475505 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.475486    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-7qnk2"
Oct 30 14:14:51.492317 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.492286    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fhsqk/perf-node-gather-daemonset-7qnk2"]
Oct 30 14:14:51.492919 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.492895    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fhsqk\"/\"kube-root-ca.crt\""
Oct 30 14:14:51.503362 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.503329    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fhsqk\"/\"openshift-service-ca.crt\""
Oct 30 14:14:51.503362 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.503335    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fhsqk\"/\"default-dockercfg-mgtrq\""
Oct 30 14:14:51.629545 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.629505    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7335a79f-64f0-424f-8bd7-30e33552dd43-podres\") pod \"perf-node-gather-daemonset-7qnk2\" (UID: \"7335a79f-64f0-424f-8bd7-30e33552dd43\") " pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-7qnk2"
Oct 30 14:14:51.629764 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.629556    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq9cw\" (UniqueName: \"kubernetes.io/projected/7335a79f-64f0-424f-8bd7-30e33552dd43-kube-api-access-bq9cw\") pod \"perf-node-gather-daemonset-7qnk2\" (UID: \"7335a79f-64f0-424f-8bd7-30e33552dd43\") " pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-7qnk2"
Oct 30 14:14:51.629764 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.629668    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7335a79f-64f0-424f-8bd7-30e33552dd43-proc\") pod \"perf-node-gather-daemonset-7qnk2\" (UID: \"7335a79f-64f0-424f-8bd7-30e33552dd43\") " pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-7qnk2"
Oct 30 14:14:51.629906 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.629802    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7335a79f-64f0-424f-8bd7-30e33552dd43-sys\") pod \"perf-node-gather-daemonset-7qnk2\" (UID: \"7335a79f-64f0-424f-8bd7-30e33552dd43\") " pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-7qnk2"
Oct 30 14:14:51.629906 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.629876    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7335a79f-64f0-424f-8bd7-30e33552dd43-lib-modules\") pod \"perf-node-gather-daemonset-7qnk2\" (UID: \"7335a79f-64f0-424f-8bd7-30e33552dd43\") " pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-7qnk2"
Oct 30 14:14:51.730844 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.730748    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7335a79f-64f0-424f-8bd7-30e33552dd43-lib-modules\") pod \"perf-node-gather-daemonset-7qnk2\" (UID: \"7335a79f-64f0-424f-8bd7-30e33552dd43\") " pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-7qnk2"
Oct 30 14:14:51.730844 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.730805    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7335a79f-64f0-424f-8bd7-30e33552dd43-podres\") pod \"perf-node-gather-daemonset-7qnk2\" (UID: \"7335a79f-64f0-424f-8bd7-30e33552dd43\") " pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-7qnk2"
Oct 30 14:14:51.730844 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.730833    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq9cw\" (UniqueName: \"kubernetes.io/projected/7335a79f-64f0-424f-8bd7-30e33552dd43-kube-api-access-bq9cw\") pod \"perf-node-gather-daemonset-7qnk2\" (UID: \"7335a79f-64f0-424f-8bd7-30e33552dd43\") " pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-7qnk2"
Oct 30 14:14:51.731112 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.730851    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7335a79f-64f0-424f-8bd7-30e33552dd43-proc\") pod \"perf-node-gather-daemonset-7qnk2\" (UID: \"7335a79f-64f0-424f-8bd7-30e33552dd43\") " pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-7qnk2"
Oct 30 14:14:51.731112 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.730904    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7335a79f-64f0-424f-8bd7-30e33552dd43-lib-modules\") pod \"perf-node-gather-daemonset-7qnk2\" (UID: \"7335a79f-64f0-424f-8bd7-30e33552dd43\") " pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-7qnk2"
Oct 30 14:14:51.731112 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.730923    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7335a79f-64f0-424f-8bd7-30e33552dd43-sys\") pod \"perf-node-gather-daemonset-7qnk2\" (UID: \"7335a79f-64f0-424f-8bd7-30e33552dd43\") " pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-7qnk2"
Oct 30 14:14:51.731112 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.730955    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7335a79f-64f0-424f-8bd7-30e33552dd43-podres\") pod \"perf-node-gather-daemonset-7qnk2\" (UID: \"7335a79f-64f0-424f-8bd7-30e33552dd43\") " pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-7qnk2"
Oct 30 14:14:51.731112 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.730978    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7335a79f-64f0-424f-8bd7-30e33552dd43-sys\") pod \"perf-node-gather-daemonset-7qnk2\" (UID: \"7335a79f-64f0-424f-8bd7-30e33552dd43\") " pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-7qnk2"
Oct 30 14:14:51.731112 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.731031    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7335a79f-64f0-424f-8bd7-30e33552dd43-proc\") pod \"perf-node-gather-daemonset-7qnk2\" (UID: \"7335a79f-64f0-424f-8bd7-30e33552dd43\") " pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-7qnk2"
Oct 30 14:14:51.751088 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.751050    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq9cw\" (UniqueName: \"kubernetes.io/projected/7335a79f-64f0-424f-8bd7-30e33552dd43-kube-api-access-bq9cw\") pod \"perf-node-gather-daemonset-7qnk2\" (UID: \"7335a79f-64f0-424f-8bd7-30e33552dd43\") " pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-7qnk2"
Oct 30 14:14:51.788146 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:51.788099    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-7qnk2"
Oct 30 14:14:52.084088 ip-10-0-134-31 kubenswrapper[5093]: W1030 14:14:52.084057    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7335a79f_64f0_424f_8bd7_30e33552dd43.slice/crio-b3ce32d0254ebda000dd6afbfd6509b6f17b07f2cbe3b740a8072ef3f909280f WatchSource:0}: Error finding container b3ce32d0254ebda000dd6afbfd6509b6f17b07f2cbe3b740a8072ef3f909280f: Status 404 returned error can't find the container with id b3ce32d0254ebda000dd6afbfd6509b6f17b07f2cbe3b740a8072ef3f909280f
Oct 30 14:14:52.084445 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:52.084413    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fhsqk/perf-node-gather-daemonset-7qnk2"]
Oct 30 14:14:52.778704 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:52.778671    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-7qnk2" event={"ID":"7335a79f-64f0-424f-8bd7-30e33552dd43","Type":"ContainerStarted","Data":"5452e28305a62bb5587ad83c50d0488ae64151b8a06cbf598ab26b6abb97b2fa"}
Oct 30 14:14:52.779153 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:52.778723    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-7qnk2" event={"ID":"7335a79f-64f0-424f-8bd7-30e33552dd43","Type":"ContainerStarted","Data":"b3ce32d0254ebda000dd6afbfd6509b6f17b07f2cbe3b740a8072ef3f909280f"}
Oct 30 14:14:52.779153 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:52.778781    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-7qnk2"
Oct 30 14:14:52.820269 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:52.820209    5093 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-7qnk2" podStartSLOduration=1.820190474 podStartE2EDuration="1.820190474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 14:14:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:14:52.819245095 +0000 UTC m=+4311.966512334" watchObservedRunningTime="2026-10-30 14:14:52.820190474 +0000 UTC m=+4311.967457717"
Oct 30 14:14:53.125136 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:53.125036    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-x8wqt_e640effb-1ff4-40f2-8d7b-53732edcd2bf/cluster-monitoring-operator/0.log"
Oct 30 14:14:53.916340 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:53.916310    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fthgj_d0385014-0783-4681-a86c-1182712e0aaa/node-exporter/0.log"
Oct 30 14:14:54.006994 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:54.006967    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-k8hb7_ebddb579-1088-4d8f-8917-e7e14883534a/server/0.log"
Oct 30 14:14:54.047488 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:54.047457    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fthgj_d0385014-0783-4681-a86c-1182712e0aaa/kube-rbac-proxy/0.log"
Oct 30 14:14:54.180928 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:54.180894    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fthgj_d0385014-0783-4681-a86c-1182712e0aaa/init-textfile/0.log"
Oct 30 14:14:55.016566 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:55.016531    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-k8hb7_ebddb579-1088-4d8f-8917-e7e14883534a/server/0.log"
Oct 30 14:14:55.247249 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:55.247208    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-hdnsr_775251e0-43d1-4239-a077-1b9fc7e56fe4/manager/0.log"
Oct 30 14:14:56.129728 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:56.129690    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-hdnsr_775251e0-43d1-4239-a077-1b9fc7e56fe4/manager/0.log"
Oct 30 14:14:58.289125 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:58.289096    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-k8hb7_ebddb579-1088-4d8f-8917-e7e14883534a/server/0.log"
Oct 30 14:14:58.796992 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:58.796958    5093 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-7qnk2"
Oct 30 14:14:59.395249 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:59.395211    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-k8hb7_ebddb579-1088-4d8f-8917-e7e14883534a/server/0.log"
Oct 30 14:14:59.465010 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:59.464973    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-hdnsr_775251e0-43d1-4239-a077-1b9fc7e56fe4/manager/0.log"
Oct 30 14:14:59.787390 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:14:59.787361    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-2w8jt_63086243-e82d-4fc5-bfef-e4241c22ad4e/networking-console-plugin/0.log"
Oct 30 14:15:00.161223 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:15:00.161025    5093 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889495-tbk9w"]
Oct 30 14:15:00.165045 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:15:00.165015    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889495-tbk9w"
Oct 30 14:15:00.181321 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:15:00.181283    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889495-tbk9w"]
Oct 30 14:15:00.190953 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:15:00.190918    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 14:15:00.191269 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:15:00.191236    5093 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 14:15:00.191419 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:15:00.191267    5093 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 14:15:00.191750 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:15:00.191686    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhvbl\" (UniqueName: \"kubernetes.io/projected/870a3ad8-6a7f-48bc-b69b-3a73d360d78e-kube-api-access-jhvbl\") pod \"maas-api-key-cleanup-29889495-tbk9w\" (UID: \"870a3ad8-6a7f-48bc-b69b-3a73d360d78e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889495-tbk9w"
Oct 30 14:15:00.191853 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:15:00.191835    5093 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/870a3ad8-6a7f-48bc-b69b-3a73d360d78e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889495-tbk9w\" (UID: \"870a3ad8-6a7f-48bc-b69b-3a73d360d78e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889495-tbk9w"
Oct 30 14:15:00.292422 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:15:00.292373    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/870a3ad8-6a7f-48bc-b69b-3a73d360d78e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889495-tbk9w\" (UID: \"870a3ad8-6a7f-48bc-b69b-3a73d360d78e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889495-tbk9w"
Oct 30 14:15:00.292619 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:15:00.292492    5093 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhvbl\" (UniqueName: \"kubernetes.io/projected/870a3ad8-6a7f-48bc-b69b-3a73d360d78e-kube-api-access-jhvbl\") pod \"maas-api-key-cleanup-29889495-tbk9w\" (UID: \"870a3ad8-6a7f-48bc-b69b-3a73d360d78e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889495-tbk9w"
Oct 30 14:15:00.293106 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:15:00.293081    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/870a3ad8-6a7f-48bc-b69b-3a73d360d78e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889495-tbk9w\" (UID: \"870a3ad8-6a7f-48bc-b69b-3a73d360d78e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889495-tbk9w"
Oct 30 14:15:00.314140 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:15:00.314086    5093 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhvbl\" (UniqueName: \"kubernetes.io/projected/870a3ad8-6a7f-48bc-b69b-3a73d360d78e-kube-api-access-jhvbl\") pod \"maas-api-key-cleanup-29889495-tbk9w\" (UID: \"870a3ad8-6a7f-48bc-b69b-3a73d360d78e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889495-tbk9w"
Oct 30 14:15:00.482393 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:15:00.482360    5093 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889495-tbk9w"
Oct 30 14:15:00.548269 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:15:00.548234    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-hdnsr_775251e0-43d1-4239-a077-1b9fc7e56fe4/manager/0.log"
Oct 30 14:15:00.790536 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:15:00.790495    5093 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889495-tbk9w"]
Oct 30 14:15:00.790536 ip-10-0-134-31 kubenswrapper[5093]: W1030 14:15:00.790514    5093 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod870a3ad8_6a7f_48bc_b69b_3a73d360d78e.slice/crio-244972cc80fae967c678c5a6fb74a7e6ae426dcc2f0d909ca961e03c1fdef6b2 WatchSource:0}: Error finding container 244972cc80fae967c678c5a6fb74a7e6ae426dcc2f0d909ca961e03c1fdef6b2: Status 404 returned error can't find the container with id 244972cc80fae967c678c5a6fb74a7e6ae426dcc2f0d909ca961e03c1fdef6b2
Oct 30 14:15:00.832508 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:15:00.832467    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889495-tbk9w" event={"ID":"870a3ad8-6a7f-48bc-b69b-3a73d360d78e","Type":"ContainerStarted","Data":"244972cc80fae967c678c5a6fb74a7e6ae426dcc2f0d909ca961e03c1fdef6b2"}
Oct 30 14:15:01.241220 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:15:01.241188    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jdj4_f93ee017-ab79-447c-90e0-5b85b71dba24/console-operator/1.log"
Oct 30 14:15:01.245633 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:15:01.245595    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jdj4_f93ee017-ab79-447c-90e0-5b85b71dba24/console-operator/2.log"
Oct 30 14:15:01.839671 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:15:01.839634    5093 generic.go:356] "Generic (PLEG): container finished" podID="870a3ad8-6a7f-48bc-b69b-3a73d360d78e" containerID="9bc2f924119e6aa4e07c9c3e572cb2436823230d2fb9a87e247d7ecc1c447819" exitCode=60
Oct 30 14:15:01.840164 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:15:01.839721    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889495-tbk9w" event={"ID":"870a3ad8-6a7f-48bc-b69b-3a73d360d78e","Type":"ContainerDied","Data":"9bc2f924119e6aa4e07c9c3e572cb2436823230d2fb9a87e247d7ecc1c447819"}
Oct 30 14:15:01.840164 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:15:01.839952    5093 scope.go:122] "RemoveContainer" containerID="9bc2f924119e6aa4e07c9c3e572cb2436823230d2fb9a87e247d7ecc1c447819"
Oct 30 14:15:02.849706 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:15:02.849665    5093 generic.go:356] "Generic (PLEG): container finished" podID="870a3ad8-6a7f-48bc-b69b-3a73d360d78e" containerID="4c1a32ad829529aff3ad9a44efda0d0a6f12fb5bdf85c0ddfc329c477a0a1f89" exitCode=60
Oct 30 14:15:02.850270 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:15:02.849750    5093 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889495-tbk9w" event={"ID":"870a3ad8-6a7f-48bc-b69b-3a73d360d78e","Type":"ContainerDied","Data":"4c1a32ad829529aff3ad9a44efda0d0a6f12fb5bdf85c0ddfc329c477a0a1f89"}
Oct 30 14:15:02.850270 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:15:02.849827    5093 scope.go:122] "RemoveContainer" containerID="9bc2f924119e6aa4e07c9c3e572cb2436823230d2fb9a87e247d7ecc1c447819"
Oct 30 14:15:02.850270 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:15:02.850197    5093 scope.go:122] "RemoveContainer" containerID="4c1a32ad829529aff3ad9a44efda0d0a6f12fb5bdf85c0ddfc329c477a0a1f89"
Oct 30 14:15:02.850574 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:15:02.850546    5093 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889495-tbk9w_odh-ai-gateway-infra(870a3ad8-6a7f-48bc-b69b-3a73d360d78e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889495-tbk9w" podUID="870a3ad8-6a7f-48bc-b69b-3a73d360d78e"
Oct 30 14:15:02.955545 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:15:02.955516    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-k8hb7_ebddb579-1088-4d8f-8917-e7e14883534a/server/0.log"
Oct 30 14:15:03.858850 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:15:03.858812    5093 scope.go:122] "RemoveContainer" containerID="4c1a32ad829529aff3ad9a44efda0d0a6f12fb5bdf85c0ddfc329c477a0a1f89"
Oct 30 14:15:03.859290 ip-10-0-134-31 kubenswrapper[5093]: E1030 14:15:03.859078    5093 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889495-tbk9w_odh-ai-gateway-infra(870a3ad8-6a7f-48bc-b69b-3a73d360d78e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889495-tbk9w" podUID="870a3ad8-6a7f-48bc-b69b-3a73d360d78e"
Oct 30 14:15:04.178284 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:15:04.178217    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-hdnsr_775251e0-43d1-4239-a077-1b9fc7e56fe4/manager/0.log"
Oct 30 14:15:04.401968 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:15:04.401929    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-lvf7h_abf08e3d-eec3-4f73-92a6-4ffb691cbd2d/volume-data-source-validator/0.log"
Oct 30 14:15:04.413192 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:15:04.413159    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-k8hb7_ebddb579-1088-4d8f-8917-e7e14883534a/server/0.log"
Oct 30 14:15:05.492345 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:15:05.492309    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-hdnsr_775251e0-43d1-4239-a077-1b9fc7e56fe4/manager/0.log"
Oct 30 14:15:06.380235 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:15:06.380205    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l_142560f0-3d00-4d0c-9f87-58746dbe8a4e/istio-proxy/0.log"
Oct 30 14:15:07.462954 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:15:07.462918    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8nb7s_0d0b2655-d52d-4c95-9589-d45237fa1057/dns/0.log"
Oct 30 14:15:07.590414 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:15:07.590376    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8nb7s_0d0b2655-d52d-4c95-9589-d45237fa1057/kube-rbac-proxy/0.log"
Oct 30 14:15:07.634568 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:15:07.634536    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l_142560f0-3d00-4d0c-9f87-58746dbe8a4e/istio-proxy/0.log"
Oct 30 14:15:07.838821 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:15:07.838744    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6f89d9f6cc5f98l_142560f0-3d00-4d0c-9f87-58746dbe8a4e/istio-proxy/0.log"
Oct 30 14:15:08.509282 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:15:08.509247    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gltng_5a16e7b2-e45d-480a-a7bc-d914640b9867/dns-node-resolver/0.log"
Oct 30 14:15:10.128817 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:15:10.128783    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-f7mvz_087058f5-e286-4ac7-8ed8-a280da332878/node-ca/0.log"
Oct 30 14:15:11.010042 ip-10-0-134-31 kubenswrapper[5093]: I1030 14:15:11.010007    5093 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-668b54766c-wgk28_d5db3ba8-f1de-4fa5-9839-e7e8200da58f/router/0.log"